Easier access to scientific data
Brought to you by NOAA NMFS SWFSC ERD
|
ERDDAP
Easier access to scientific data |
Brought to you by NOAA NMFS SWFSC ERD |
ERDDAP is a great example of
User-Driven Innovation
,
where product innovation often comes from consumers (ERDDAP users), not just the producers (ERDDAP developers).
Over the years, most of the ideas for new features and changes in ERDDAP have come from users.
Those users are credited below for their great ideas. Thank you! Please keep
those great suggestions coming!
Here are the changes associated with each ERDDAP release.
My impression of Java 17 is that it prefers more processing power and memory
for long-running, larger applications like ERDDAP, so it works slightly slower
than Java 8 with low power computers (e.g., 2 cores and minimal RAM)
and works slightly faster than Java 8 with higher power computers
(e.g., 4+ cores and plentiful RAM).
So if you see poor performance, use programs like Linux's
top
to check resource usage and consider giving ERDDAP more resources.
Memory is cheap! Most phones have more processors and memory than
the servers that some of you are using to run ERDDAP!
Thanks to Erin Turnbull.
has replaced AdoptOpenJDK as the main/recommended source of Java (OpenJDK).
Not all text or all web pages were translated. There were time constraints on this project that prevented Qi and Bob from getting to 100%.
The obvious question is: why did we put so much effort into this when Chrome will translate web pages on-the-fly? The answer is: this way, we get much more control over how the translation is done. Notably, there are lots of words that shouldn't be translated on the web pages, e.g., the titles and summaries of datasets, the names of variables, parameters, units, and organizations. Much of the translation effort was identifying words and phrases that shouldn't be translated. Also, the machine translations tended to mangle certain types of HTML markup. Managing the translation allowed us to minimize this problem.
The translation project was done by Qi Zeng (a Google Summer of Code intern) and Bob Simons using Google's Translation web service. It was a huge project. Thanks, Qi!
<theShortDescriptionHtml><![CDATA[ <h1>ERDDAP</h1> &erddapIs; &thisParticularErddap; [standardShortDescriptionHtml] ]]></theShortDescriptionHtml>The last 3 lines of content are things that will be substituted with translated text. If you convert any of them (notably &thisParticularErddap;) or all of them to explicit text in datasets.xml (which has priority, if present) or messages.xml, that text will appear no matter what language the user selects. This is not perfect, but I figured that few administrators would want to edit <theShortDescriptionHtml> in 35 different files to provide 35 different translated versions of that tag.
<ipAddressMaxRequests></ipAddressMaxRequests> <!-- current default=7 --> <ipAddressMaxRequestsActive></ipAddressMaxRequestsActive> <!-- current default=2 --> <ipAddressUnlimited></ipAddressUnlimited> <!-- default=empty -->For further information, see ipAddressMaxRequests. ERDDAP also now prints the "Number of unique users (since startup)" on the status.html page.
It is possible that this tag will lead to other, similar diagnostic information being emailed to
erd.data at noaa.gov in the future.
The email's content will always be minimal and related to bugs, and not, for example, usage information.
Thanks to Marco Alba.
RECOMMENDED (because there are users who want this):
If you want to make all EDD...FromFiles datasets accessible via the files system,
then
The latest version of Java 8 (not Java 9, 10, 11, ...) from
AdoptOpenJDK
remains the recommended version of Java for ERDDAP.
Java 8 has Long Term Support from AdoptOpenJDK so it remains safe to use,
but remember to get the latest version of it periodically for security reasons.
WARNING: For datasets that are affected by this change, it is possible that you will
see problems with the dataset because the data that ERDDAP reads from the source may be
different (e.g., variables previously read as signed integers may now be read as unsigned integers).
The resulting problems include: new files not being added to the dataset,
and/or errors when you try to access the data.
If a dataset has problems, the first thing to try is to
set a hardFlag for the dataset.
If that doesn't resolve the problem, then you have to look at log.txt to see the
error messages, delve into the datasets.xml for the dataset,
and/or perhaps rerun generateDatasets.xml for the dataset.
Thanks to netcdf-java 5.x (which forced the issue) and the upcoming CF 1.9.
WARNING: It is possible this will cause problems for some existing datasets (e.g., cause new files to be labeled "bad"). If so, set a hardFlag for the dataset so that all of the source files will be reread with the new system.
,
you can edit the existing rules or add new rules.
See ERDDAP-lint
for more information.
This tool is especially useful for datasets that you created some time ago and now want to bring up-to-date with your current metadata preferences. For example, early versions of GenerateDatasetsXml didn't put any effort into creating global creator_name, creator_email, creator_type, or creator_url metadata. You could use ERDDAP-lint to identify the datasets that lack those metadata attributes.
Thanks to Rob and Adam for creating this tool and making it available to the ERDDAP community.
There are also new statistics named "Large Request, IP address" which show the IP addresses of the users who made large requests (currently, gridded .nc files > 1GB).
Also, the time series table on the status.html page now includes a "memFail"
column showing the number of requests that failed with "OutOfMemory (Too Big)"
errors since the last major Load Datasets. Any number other than 0 here is
at least some cause for concern.
Thanks to Bob Simons.
.
If you click on the "files" link for any of those sample datasets,
you can browse the directory tree and files in that S3 bucket.
Because of the way these datasets work, these
directory listings are always perfectly up-to-date because ERDDAP gets them on-the-fly.
If you click down the directory tree to an actual file name and click on the file name,
ERDDAP will redirect your request to AWS S3 so that you can download the file
directly from AWS.
ERDDAP administrators can
I explored a different solution (storing strings in StringArray as UTF-8-encoded byte arrays). That reduces memory usage another ~33%, but at the cost of ~33% slowdown. Compared to the system that is now being used, that seemed like a bad trade off. It's easier to give a computer more memory (buy more memory for ~$200) than to make it faster (buy a whole new computer).
If it is convenient, it's still always a good idea to split huge tabular data files into several smaller files based on some criteria like stationID and/or time. ERDDAP will often only have to open one of the small files in response to a user's request, and thus be able to respond much faster.
I didn't clearly state before: you should just use standardizeWhat features when you actually need them (e.g., when different source files store time values in different ways), because some requests to datasets that use standardizeWhat will be processed a little slower.
account.
<!-- The tags below are described in setupDatasetsXml.html.
The defaults listed below are as of ERDDAP v2.00. -->
<cacheMinutes></cacheMinutes> <!-- default=60 -->
<decompressedCacheMaxGB></decompressedCacheMaxGB> <!-- default=10 -->
<decompressedCacheMaxMinutesOld></decompressedCacheMaxMinutesOld> <!-- default=15 -->
<drawLandMask></drawLandMask> <!-- "over" or "under" (default) -->
<graphBackgroundColor></graphBackgroundColor> <!-- 0xAARRGGBB, default is 0xffccccff -->
<loadDatasetsMinMinutes></loadDatasetsMinMinutes> <!-- usually=default=15 -->
<loadDatasetsMaxMinutes></loadDatasetsMaxMinutes> <!-- default=60 -->
<logLevel></logLevel> <!-- "warning" (fewest messages), "info" (default), or "all" (most messages) -->
<nGridThreads></nGridThreads> <!-- default=1 -->
<nTableThreads></nTableThreads> <!-- default=1 -->
<partialRequestMaxBytes></partialRequestMaxBytes> <!-- default=490000000 -->
<partialRequestMaxCells></partialRequestMaxCells> <!-- default=10000000 -->
<slowDownTroubleMillis></slowDownTroubleMillis> <!-- default=1000 -->
<unusualActivity></unusualActivity> <!-- default=10000 -->
<!-- The defaults for the following tags are in messages.xml. -->
<startHeadHtml5></startHeadHtml5>
<startBodyHtml5></startBodyHtml5> <!-- This is often customized. -->
<theShortDescriptionHtml></theShortDescriptionHtml> <!-- This is often customized. -->
<endBodyHtml5></endBodyHtml5>
<standardLicense></standardLicense>
<standardContact></standardContact>
<standardDataLicenses></standardDataLicenses>
<standardDisclaimerOfEndorsement></standardDisclaimerOfEndorsement>
<standardDisclaimerOfExternalLinks></standardDisclaimerOfExternalLinks>
<standardGeneralDisclaimer></standardGeneralDisclaimer>
<standardPrivacyPolicy></standardPrivacyPolicy>
If your value is different from the new suggested default (other than for <startBodyHtml5> and <theShortDescriptionHtml>, which are useful for customizing your ERDDAP installation), please consider switching to the new default values. This is particularly true of <partialRequestMaxBytes> and <partialRequestMaxCells>, where the default/suggested value has changed significantly over the years.
After you copy each value, delete the tag and its description from setup.xml. It is better to have these tags in datasets.xml. And there are now better descriptions in setupDatasetsXml.html.
A quirk of the new system is that the very first web page when you start up ERDDAP will be the default ERDDAP web page. Every subsequent webpage will use the ...Html content you specify in datasets.xml.
.
because they meet the DOI requirement
that the dataset is unchanging, except by aggregation.
See
EDDTableFromHttpGet.
Thanks to OOI (from long ago and now) for talking about the need for this
and Eugene Burger for the reminder about working on what is important.
This works great!
In most cases, the slowdown related to decompressing the
files is minor. We strongly encourage you to try this, notably for datasets
and/or data files that are infrequently used.
This may save you $30,000 or more!
This is one of the few ERDDAP features that can save you lots of money --
if you compress a lot of data files, you will need far fewer RAIDs/hard drives
to store the data, or conversely, you can serve far more data (up to 10x) with the
RAIDs you already have. If this feature saves you from buying another RAID,
then it has saved you about $30,000.
See the Externally Compressed Files documentation. Thanks to Benoit Perrimond and Paloma de la Vallee.
("Better than CSV").
Thanks to the people at the Marine Institute of Ireland for telling me about this format
and to Eugene Burger and PMEL for the request to support it as an input type.
,
in the context of a Dataset, were more specific.
Software developers should be able to write an implementation of a specification
based on the specification alone, without guidance from experts.
But I defer to Google (notably Natasha Noy), NCEI (notably John Relph), and Rob Fuller.
specified the correct order.
Software developers should be able to write an implementation of a specification
based on the specification alone, without guidance from experts.
Thanks to Natasha Noy and Rob Fuller.
-- isn't an ERDDAP feature, but will be of interest to many ERDDAP users.
Erddapy (ERDDAP + Python) is a Python library created by Filipe Fernandes that
"takes advantage of ERDDAP's RESTful web services and creates the
ERDDAP URL for any request like searching for datasets, acquiring metadata,
downloading data, etc." Thanks to Filipe Fernandes.
.
Thanks to
rOpenSci
and Roy Mendelssohn.
We recommend using the default <startHeadHtml5> and <endBodyHtml5>.
We recommend: If you made changes to the original
<startBodyHtml> and/or want to customize
your ERDDAP now, please copy the new <startBodyHtml5> tag (from below)
into your setup.xml and modify it to customize your ERDDAP so that
ERDDAP's web pages reflect your organization, not NOAA ERD.
Notably, please change the "Brought to you by" to your organization(s).
If you need help, please email erd.data at noaa.gov.
(If you don't want to customize your ERDDAP now, use the default <startBodyHtml5>.)
Then delete the 3 old tags in your setup.xml which are no longer used.
<startBodyHtml5><![CDATA[
<body>
<table class="compact nowrap" style="width:100%; background-color:#128CB5;">
<tr>
<td style="text-align:center; width:80px;"><a rel="bookmark"
href="https://www.noaa.gov/"><img
title="National Oceanic and Atmospheric Administration"
src="&erddapUrl;/images/noaab.png" alt="NOAA"
style="vertical-align:middle;"></a></td>
<td style="text-align:left; font-size:x-large; color:#FFFFFF; ">
<strong>ERDDAP</strong>
<br><small><small><small>Easier access to scientific data</small></small></small>
</td>
<td style="text-align:right; font-size:small;">
&loginInfo;
<br>Brought to you by
<a title="National Oceanic and Atmospheric Administration" rel="bookmark"
href="https://www.noaa.gov">NOAA</a>
<a title="National Marine Fisheries Service" rel="bookmark"
href="https://www.fisheries.noaa.gov">NMFS</a>
<a title="Southwest Fisheries Science Center" rel="bookmark"
href="https://www.fisheries.noaa.gov/about/southwest-fisheries-science-center">SWFSC</a>
<a title="Environmental Research Division" rel="bookmark"
href="https://www.fisheries.noaa.gov/about/environmental-research-division-southwest-fisheries-science-center">ERD</a>
</td>
</tr>
</table>
]]></startBodyHtml5>
There are additional ways you can customize ERDDAP so ERDDAP's web pages reflect your organization instead of NOAA ERD.
If you didn't customize those tags, please delete them from your setup.xml file.
Now they all have defaults in messages.xml that refer to datasets
in Bob's ERDDAP at https://coastwatch.pfeg.noaa.gov/erddap/index.html .
So you no longer need to have specific datasets in your ERDDAP.
If you want to override the defaults, copy some or all of those tags
into your setup.xml and change their values.
If you want the examples to point to your ERDDAP, the easiest method is:
<dataset type="EDDGridFromErddap" datasetID="jplMURSST41" active="true">
<sourceUrl>https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41</sourceUrl>
</dataset>
<dataset type="EDDTableFromErddap" datasetID="pmelTaoDySst" active="true">
<sourceUrl>https://coastwatch.pfeg.noaa.gov/erddap/tabledap/pmelTaoDySst</sourceUrl>
</dataset>
<EDDGridErddapUrlExample>https://coastwatch.pfeg.noaa.gov/erddap/</EDDGridErddapUrlExample> <EDDTableErddapUrlExample>https://coastwatch.pfeg.noaa.gov/erddap/</EDDTableErddapUrlExample>
If you did customize those tags, leave them as is and please add these 2 new tags to your setup.xml to specify the ERDDAP URL for these datasets, but change the URL to your ERDDAP's (https?) URL:
<EDDGridErddapUrlExample>https://coastwatch.pfeg.noaa.gov/erddap/</EDDGridErddapUrlExample> <EDDTableErddapUrlExample>https://coastwatch.pfeg.noaa.gov/erddap/</EDDTableErddapUrlExample>
requests for portions of /files/ files. This was needed to support
the audio and video viewers in browsers.
So, please check your datasets: if a variable's values are packed and
if actual_range is specified as packed data values,
please add an <addAttributes> actual_range value to specify the unpacked values.
Otherwise, the dataset will not load in ERDDAP.
A simple and almost perfect way to do this is to search your datasets.xml for
sourceAttributes that have
<att name="actual_range" type="shortList">
or <att name="actual_range" type="intList">
and a scale_factor other than 1.0.
Those are the actual_range attributes that you might have to fix.
For axis variables in EDDGrid datasets, ERDDAP always sets the actual_range attribute to be the actual range of the values since it knows those values.
For axis variables with descending values (e.g., some latitude variables), ERDDAP created actual_range with the [0]...[last] values, which were high...low. Now it always uses low...high values to make the new CF definition.
The correctness of the actual_range values is particularly important for EDDTable datasets, because ERDDAP will quickly reject user requests for data values which are less than the actual_range minimum value or which are greater than the actual_range maximum value.
Related: the actual_min, actual_max, data_min and data_max attributes are now deprecated. Please convert your datasets to use actual_range instead.
to make your data catalog and datasets part of the
semantic web,
which is Tim Berners-Lee's idea to make web content more machine readable and
machine "understandable".
Search engines
(Google in particular)
and other semantic tools can use this structured markup
to facilitate discovery and indexing.
The json-ld structured markup appears as invisible-to-humans <script> code on the
http://.../erddap/info/index.html web page (which is a semantic web
DataCatalog)
and on each http://.../erddap/info/datasetID/index.html web page
(which is a semantic web
Dataset).
(Special thanks to Adam Leadbetter and Rob Fuller of the Marine Institute in Ireland
for doing the hard parts of the work to make this part of ERDDAP.)
The netcdf-java developers maintain that these vulnerabilities are not relevant
because of the way that netcdf code uses these libraries
and in any case would only be relevant when accessing Amazon S3.
See
https://github.com/Unidata/thredds/issues/866 .
I believe them. If you still have concerns about this,
please contact the netcdf-java developers.
(Note that if you don't believe the netcdf-java developers
and are contemplating not using ERDDAP because of this, you shouldn't use
THREDDS either, because THREDDS uses netcdf-java more fundamentally and
more extensively than ERDDAP.)
Details: The troublesome code and the vulnerability warnings are:
netcdfAll-latest.jar/META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml
See https://nvd.nist.gov/vuln/detail/CVE-2016-7051 -- High
netcdfAll-latest.jar/META-INF/maven/com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/pom.xml
See https://nvd.nist.gov/vuln/detail/CVE-2016-7051 -- High
netcdfAll-latest.jar/META-INF/maven/com.fasterxml.jackson.core/jackson-annotations/pom.xml
See https://nvd.nist.gov/vuln/detail/CVE-2016-7051 -- High
See https://nvd.nist.gov/vuln/detail/CVE-2016-3720 -- Critical
netcdfAll-latest.jar/META-INF/maven/com.fasterxml.jackson.core/jackson-core/pom.xml
See https://nvd.nist.gov/vuln/detail/CVE-2016-7051 -- High
See https://nvd.nist.gov/vuln/detail/CVE-2016-3720 -- Critical
"For version 4.6.10, aws-java-sdk-core pulls in version 2.6.6 of jackson-* artifacts."
(email from netcdf-java people).
Thanks to Charles Carleton and NCEI.
.
)
where each line has a separate JSON object.
.jsonlCSV just has the values in a CSV format.
.jsonlKVP has Key:Value pairs.
Each line stands on its own. The lines are not enclosed in a larger JSON array or object.
For an example, see
this sample request.
Thanks to Damian Smyth, Rob Fuller, Adam Leadbetter, and Ireland's Marine Institute.
.
NCEI may standardize on this format.
Thanks to Scott Cross and John Relph.
-like
printout of the header of an .nc file.
You can also print the data values for specified variables
(or enter "nothing" to not print any data values).
This is useful because, without ncdump it is hard to know what is in
a file and thus which EDDType you should specify for GenerateDatasetsXml.
Thanks to Craig Risien, Rich Signell, Christopher Wingard and OOI.
<att name="colorBarMaximum" type="double">8000.0</att>
<att name="colorBarMinimum" type="double">-8000.0</att>
<att name="colorBarPalette">TopographyDepth</att>
<!-- If you want to restrict access to some datasets, you need to specify the method used for logging on (authentication). See the info at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security Currently, the options are: "" (logins not supported, the default), "custom", "email", and "google" (recommended). [No longer supported: "basic", "openid"] -->
<!-- If authentication=google, you must supply your Google Client ID. See https://developers.google.com/identity/sign-in/web/devconsole-project When setting this up, for Authorized JavaScript origins, for testing on your computer, use the domain "localhost" (e.g., origin=https://localhost:8443), not "127.0.0.1" (because Google Sign-In doesn't work with anything at that domain). This will be a string of about 75 characters, probably starting with several digits and ending with .apps.googleusercontent.com . --> <googleClientID></googleClientID>
In your setup.xml, please change the description for <baseUrl> to be
<!-- baseUrl is the start of the public URL, to which "/erddap" is appended. For example: For running/testing on your personal computer: <baseUrl>http://localhost:8080</baseUrl> (127.0.0.1 doesn't work with authentication=google). If you want to encourage all users to use https (not http), make the baseUrl the same as the baseHttpsUrl (see below). For ERD releases, we used to use <baseUrl>http://coastwatch.pfeg.noaa.gov</baseUrl> For ERD releases, we now use <baseUrl>https://coastwatch.pfeg.noaa.gov</baseUrl> -->
<!-- For "custom" authentication, this specifies how you have stored passwords in the roles tags in datasets.xml. If you aren't storing any passwords, this is irrelevant. The options (in order of increasing security) are: "MD5", "UEPMD5" (MD5(UserName:ERDDAP:Password)), "SHA256", "UEPSHA256" (SHA256(UserName:ERDDAP:Password), the default). You should only use "MD5" or "SHA256" if you need to match values stored that way in an external password database. See the info at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security -->
<!-- This is a variant of <baseUrl> which is used when authentication is active and the user is logged in. In general, you take the <baseUrl>, change "http" to "https", and change/add ":8443". This must begin with "https://". If you make a proxy so that ":8443" isn't needed, then don't use ":8443" here. This is relevant even if <authentication> is "". See the instructions at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security For example: For running/testing on your personal computer: <baseHttpsUrl>https://localhost:8443</baseHttpsUrl> For releases at ERD, we use: <baseHttpsUrl>https://coastwatch.pfeg.noaa.gov</baseHttpsUrl> If you want to encourage all users to use https (not http), make the baseUrl (see above) the same as the baseHttpsUrl. -->
<!-- Normally, if you have a EDDGridFromErddap or EDDTableFromErddap dataset in your datasets.xml, it will try to subscribe to the remote ERDDAP dataset so that the local dataset is kept perfectly up-to-date. If this ERDDAP is not publicly accessible (http://localhost), or its IP address will change soon, or you have some other reason, you can tell this ERDDAP to not try to subscribe to the remote ERDDAP datasets by setting this to false. (default=true) This is the overall setting for this ERDDAP. It can be overridden by the same tag (with a different value) in the datasets.xml chunk for a given EDD...FromErddap dataset. For each fromErddap dataset that doesn't subscribe to the remote ERDDAP dataset, you should set <reloadEveryNMinutes> to a smaller number so that the local dataset stays reasonably up-to-date. --> <subscribeToRemoteErddapDataset>true</subscribeToRemoteErddapDataset>
(no more security updates) in April 2015.
This version of ERDDAP will not work with versions of Java below 1.8.
If you update from Java 1.7x (or earlier), you should also update Tomcat.
See the ERDDAP Set Up Instructions for download links and advice.
in the request header. THREDDS and Amazon S3 support Range Requests,
Hyrax does not.
This system allows you to access data in remote files without downloading the files
(which is helpful if the remote files are too voluminous),
but access to these files will be far slower than access to local files
or even to a remote OPeNDAP source.
.
,
for example, plotting longitude versus depth, or time versus depth.
[Note: if depth is on the Y Axis, it will probably be flipped from what you want.
Sorry, un-flipping it is not yet an option.]
Thanks to Cara Wilson and Lynn DeWitt.
.
Thanks to Ocean Networks Canada.
We still recommend: If a dataset has a large number of files (e.g., >1,000), the operating system (and thus EDDGridFromFiles and EDDTableFromFiles) will operate much more efficiently if you store the files in a series of subdirectories (one per year, or one per month for datasets with very frequent files), so that there are never a huge number of files in a given directory.
-formatted
description of the dataset (similar to a combined .dds + .das).
.
if you have translated messages.xml).
Multidimensional Array options,
and which therefore conforms to the NODC templates
[2021: now the NCEI templates]
for storing this type of data.
Thanks to NODC.
.
For more information, see the
Wiki
for the ERDDAP project as well as the more general
ERDDAP Programmer's Guide.
(This was announced separately a few weeks after the ERDDAP 1.38 release.)
conventions. Thanks to NODC and
special thanks to Kyle Wilcox for making sample files for the huge number
of valid DSG file formats
and for making them publicly available.
<!-- If true, when you start up ERDDAP, some types of datasets (e.g.,
EDDGridFromDap) will used cached information (.dds, .das, etc.) to reload
very quickly, without contacting the remote server. The dataset's age
will be based on when the dataset was reloaded last. Normally this
should be true (the default), but set it to false if you want to bypass
the cached information.
<quickRestart>true</quickRestart>
<!-- ERDDAP lets you choose between two search engines for full text searches:
* original (the default) -- is the best choice if your ERDDAP has fewer
than about 10,000 datasets. It is very robust and trouble free.
* lucene -- is the best choice for more than about 10,000 datasets.
The advantages are that with any number of datasets it works fast
and uses very little memory.
But there are many things that might go wrong with individual
queries and with the whole system.
And although its behaviour (the datasets it finds and the order that
it ranks them) is almost identical to the original search engine,
it has a few quirky, subtle, small differences.
-->
<searchEngine>original</searchEngine>
.
MySql and PostgreSQL seem to have better defaults for this setting.
Thanks to Kevin O'Brien.
<!-- Tabledap Examples
This group of settings is used to make examples for the tabledap documentation
that appears at [baseUrl]/erddap/tabledap/documentation.html and elsewhere.
If you include the erdGlobecBottle dataset in your datasets.xml (recommended),
you don't need to change these.
If you don't, you MUST change these before you make your ERDDAP public;
otherwise, none of the examples will work!
The new settings should be very similar to the defaults.
If your ERDDAP won't serve any tabular datasets, use "NOT_APPLICABLE" for all of the entities.
In .xml files like this, ampersand, lessThan, and greaterThan have to be
HTML encoded as "&", "<", ">".
-->
<!-- This is the datasetID for an EDDTable dataset that is served by your ERDDAP.
This dataset is used as the basis for all of the EDDGrid examples below.
Ideally, it is a dataset that has longitude, latitude, and time variables (among others).
('time' allows for making a time series graph. 'latitude' and 'longitude' allow for making a map.)
The dataset can have longitude values -180 to 180, or 0 to 360. -->
<EDDTableIdExample>erdGlobecBottle</EDDTableIdExample>
<!-- This is a comma-separated list of variables from the dataset.
It is useful if it is "longitude,latitude,time," plus a data variable name. -->
<EDDTableVariablesExample>longitude,latitude,time,bottle_posn,temperature1</EDDTableVariablesExample>
<!-- This is the constraints example which is appended to EDDTableVariablesExample. -->
<EDDTableConstraintsExample>&time>=2002-08-17T00:00:00Z&time<=2002-08-19T20:18:00Z</EDDTableConstraintsExample>
<!-- This is an example data query using an ISO-formatted time.
You could generate your example via your dataset's Data Access Form in ERDDAP. -->
<EDDTableDataTimeExample>longitude,latitude,time,bottle_posn,temperature1&time>=2002-08-17T00:00:00Z&time<=2002-08-19T20:18:00Z</EDDTableDataTimeExample>
<!-- This is an equivalent example data query, but which specifies time as seconds-since-1970-01-01.
If you need to convert a date/time to "seconds since 1970-01-01", use
https://coastwatch.pfeg.noaa.gov/erddap/convert/time.html -->
<EDDTableDataValueExample>longitude,latitude,time,bottle_posn,temperature1&time>=1029542400&time<=1029788280</EDDTableDataValueExample>
<!-- This is an example query which generates a graph.
You could generate your example via your dataset's Make A Graph form in ERDDAP. -->
<EDDTableGraphExample>bottle_posn,temperature1&time=2002-08-19T10:06:00Z&.draw=lines</EDDTableGraphExample>
<!-- This is an example query which generates a map.
In the default mapExample, temperature1, time, bottle_posn are useful
because they appear in GoogleEarth with the .kml example
and are ignored by the other image file types. -->
<EDDTableMapExample>longitude,latitude,temperature1,time,bottle_posn&time>=2002-08-13T00:00:00Z&time<=2002-08-20T00:00:00Z&bottle_posn=1&.draw=markers&.marker=5|5</EDDTableMapExample>
<!-- This is a Matlab example which uses data from the EDDTableGraphExample.
Note the Matlab notation datasetName.variableName. -->
<EDDTableMatlabPlotExample>plot(erdGlobecBottle.bottle_posn, erdGlobecBottle.temperature1)</EDDTableMatlabPlotExample>
Note that actual_range is unchanged: it may have low,high values or high,low values, since it is intended to indicate the range and the order of storage.
.
Perhaps someday soon, that Word file will be converted to .html and replace
the current "OBSOLETE" information on that web page. Thanks to the NOAA UAF project.
.
These files are structured to reflect the CDM data type of the dataset.
Since the proposed conventions just changed, as of this writing, the netcdf-java
library does not yet support reading the file formats created by ERDDAP
and interpreting them as CDM data files. It probably will soon.
Thanks to the NOAA UAF project.
units names.
This only affects units *names*, not data values. Thanks to NOAA IOOS SOS.
.
<!-- These default accessConstraints, fees, and keywords are used by the SOS, WCS, and WMS services. They can be overridden by "accessConstraints", "fees", "keywords" attributes in a dataset's global metadata. If a dataset that has an "accessibleTo" tag doesn't override "accessConstraints", then the default for "accessConstraints" is the "accessRequiresAuthorization" value. --> <accessConstraints>NONE</accessConstraints> <accessRequiresAuthorization>only accessible to authorized users</accessRequiresAuthorization> <fees>NONE</fees> <keywords>Earth science, oceans</keywords> <!-- This appears on the erddap/legal.html web page after the General Disclaimer. You can replace any of the [standardParts] with your own HTML. --> <legal><![CDATA[ [standardDisclaimerOfEndorsement] [standardDisclaimerOfExternalLinks] [standardPrivacyPolicy] [standardDataLicenses] ]]></legal> <!-- Specify the default units standard (e.g., "UDUNITS" (the default) or "UCUM") that you (the ERDDAP admin) are using to specify units. The value is case-sensitive. This is used by ERDDAP's SOS server to determine if the units need to be converted to UCUM units for WMS and SOS GetCapabilities responses. --> <units_standard>UDUNITS</units_standard> <!-- For the wms examples, pick one of your grid datasets that has longitude and latitude axes. The sample variable must be a variable in the sample grid dataset. The bounding box values are minx,miny,maxx,maxy. --> <wmsSampleDatasetID>erdBAssta5day</wmsSampleDatasetID> <wmsSampleVariable>sst</wmsSampleVariable> <!-- The bounding box values are minLongitude,minLatitude,maxLongitude,maxLatitude. Longitude values within -180 to 180, or 0 to 360, are now okay. --> <wmsSampleBBox>0,-75,360,75</wmsSampleBBox>
<!-- startHeadHtml has the start of the HTML document and the 'head' tags (starting at "<!DOCTYPE>", but not including "</head>") for all HTML web pages. This may include &erddapUrl;, which is expanded to be [baseUrl]/erddap (or [baseUttpsUrl]/erddap if the user is logged in). If your ERDDAP allows users to log in, all referenced image files, css files, etc. must be in [tomcat]/content/erddap/images or a subdirectory and must be referenced here with &erddapUrl;/images/[fileName]. favicon.ico is the image that browsers associate with your website. For more information, see https://en.wikipedia.org/wiki/Favicon . You can use your own favicon.ico file by putting it in [tomcat]/content/erddap/images. *** Optional: you can change the appearance of all of your ERDDAP's HTML pages by changing the CSS <style> settings below. For an example of a very different style, change the import reference to <tomcat>/content/erddap/images/erddapAlt.css *** If your CSS style includes links to files (e.g., images), that style information must be inline in the style tag below, after the 'import' line, not in the .css file. Put all of the (e.g., image) files in the [tomcat]/content/erddap/images directory (or a subdirectory) and reference them below starting with &erddapUrl;. Why? On ERDDAP https: web pages, *all* links should use "https:" (not "http:"); otherwise, most browsers consider the web page not fully secure. Because ERDDAP would use the same .css file for http: and https: web pages, the links within the .css file wouldn't switch between http: and https:. There doesn't seem to be a way around this other than using inline style information. --> <startHeadHtml><![CDATA[ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <html> <head> <title>ERDDAP</title> <link rel="shortcut icon" href="&erddapUrl;/images/favicon.ico"> <style type="text/css"> <!-- @import "&erddapUrl;/images/erddap.css"; --> </style> ]]></startHeadHtml> <!-- The tableCommonBGColor MUST be the same color as the table.commonBGColor in erddap.css above. Suggested is #f1ecd8. But if you use erddapAlt.css, change this to #e7dec5. --> <tableCommonBGColor>#f1ecd8</tableCommonBGColor> <!-- This is used, e.g., for the type=variable rows on the metadata info tables. --> <tableHighlightBGColor>#cceecc</tableHighlightBGColor>Thanks to POST, Hans Vedo, and Rick Blair.
<!-- The end of the body of the HTML code for all HTML web pages
(with "</body>" at the end).
This may include &erddapUrl;, which is expanded to be
[baseUrl]/erddap (or [baseUttpsUrl]/erddap if the user is logged in).
If your ERDDAP allows users to log in, all referenced image files, etc.
must be in [tomcat]/content/erddap/images or a subdirectory
and must be referenced here with &erddapUrl;/images/[fileName].
You can change this, but please keep "ERDDAP, Version &erddapVersion;"
and these references to the Disclaimers and Privacy Policy. -->
<endBodyHtml><![CDATA[
<br>
<hr>
ERDDAP, Version &erddapVersion;
<br><font class="subduedColor">Questions, comments,
suggestions? Please send an email to
<tt>erd dot data at noaa dot gov</tt>
<br>and include the ERDDAP URL directly related to your question
or comment.
<br>
<a href="&erddapUrl;/legal.html">Disclaimers</a> |
<a href="&erddapUrl;/legal.html#privacyPolicy">Privacy
Policy</a>
</font>
</body>
]]></endBodyHtml>
<theShortDescriptionHtml><![CDATA[ <h1>ERDDAP</h1> This website (the Environmental Research Division's Data Access Program) aggregates scientific data from diverse local and remote sources and offers you a simple, consistent way to download subsets of the data in common file formats and make graphs and maps. This particular ERDDAP installation has oceanographic data (for example, data from satellites and buoys). [standardShortDescriptionHtml] ]]></theShortDescriptionHtml>Feel free to change this, particularly the last sentence in the first paragraph.
).
sourceNeedsExpandedFP_EQ is set to true by default (the safest setting),
so you don't need to make
any changes.
<!-- If you want to restrict access to some datasets, you need to specify the method used for logging on (authentication). See the info at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security Currently, the options are: "" (logins not supported, the default), "custom", "openid". Note that openid login doesn't work when testing with localhost (https://127.0.0.1:8443). --> <authentication></authentication> <!-- This specifies how you have stored passwords in the roles tags in datasets.xml. If you aren't storing any passwords this is irrelevant. The options (in order of increasing security) are: "plaintext", "MD5", or "UEPMD5" (MD5(UserName:ERDDAP:Password), the default). You should only use "plaintext" or "MD5" if you need to match values stored that way in an external password database. See the info at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security --> <passwordEncoding>UEPMD5</passwordEncoding> <!-- This determines whether datasets that the user doesn't currently have access to (because he isn't logged in or because his roles don't allow access) should be shown on lists of data sets (e.g., from full text search, categorize, view all datasets, ...). The options are: "true", or "false" (the default). If false, no information about the dataset (even its existence) is shown to users who don't have access to it. If true, some information about the dataset (title, summary, etc) is shown to users who don't have access to it. If the user clicks on a link to a dataset he doesn't have access to, he will get an error message and be prompted to log in. --> <listPrivateDatasets>false</listPrivateDatasets> <!-- If the number of requests between two runs of LoadDatasets exceeds unusualActivity, an email is sent to emailEverythingTo. The default is 10000. --> <unusualActivity>10000</unusualActivity>
There is nothing new about making a local copy of a dataset. What is new here is that these classes make it *easy* to create and *maintain* a local copy of data from a *variety* of types of remote data sources and *add metadata* while copying the data.
These dataset types are part of a complete set of features that simplify the creation of grids/clusters/federations of ERDDAPs to handle very heavy loads (e.g., in a data center).
request
by adding "&.jsonp=functionName" to the end of the query.
Basically, this just tells ERDDAP to add "functionName(" to the beginning of the response and
")" to the end of the response.
If originally there was no query, leave off the "&" in your query.
Thanks to Greg Williams.
<!-- ERDDAP has a service that lets remote users set a flag to notify ERDDAP to try to reload a dataset. These requests use a key which is generated based on baseUrl/warName, a datasetID, and flagKeyKey. *** Change this once, to any text (a favorite quote? random text? It doesn't matter). Normally, you won't ever change this again. But if you think someone is abusing the flag system, change this text again, restart ERDDAP, and send all of the users of the flag system the relevant new flagKeys (see the list in the Daily Report). --> <flagKeyKey>A stitch in time saves nine. CHANGE THIS!!!</flagKeyKey> <!-- ERDDAP has an email/URL subscription system which sends a user an email or pings a url whenever a dataset of interest changes. (This is different from the RSS system, which is always active.) The system relies on the server being able to send out emails to people to validate their subscription requests. The emails appear to come from the emailFromAddress below. So if your server can't send out emails, don't make this system active. You may choose (for whatever reason) to make this system active or not, so valid values below are "true" (the default) and "false". Note that if you change this and restart ERDDAP, the list of subscriptions (in [bigParentDirectory]/subscriptionsV1.txt) isn't affected. See also the subscriptionEmailBlacklist in datasets.xml. --> <subscriptionSystemActive>true</subscriptionSystemActive>
<!-- The bounding box values are minLongitude,minLatitude,maxLongitude,maxLatitude. Longitude values within -180 to 180, or 0 to 360, are now okay. --> <wmsSampleBBox>0,-75,360,75</wmsSampleBBox>
to all files requested from the
/images directory.
This greatly reduces the number of static file requests sent to ERDDAP
and thus greatly speeds up most ERDDAP page loads.
Also, many JavaScript file references moved to the bottom of their HTML pages,
which also speeds up many ERDDAP page loads.
Thanks to the book "High Performance Web Sites" by Steve Souders and the ySlow addition to the
FireBug plugin in FireFox.
<!-- drawLand specifies the default Make A Graph setting for whether the landmask should be drawn "over" (the default) or "under" surface data on maps. "over" is recommended for primarily oceanographic data (so that grid data over land is obscured by the landmask). "under" is recommended for all other data. --> <drawLand>over</drawLand> <!-- Information about the ERDDAP administrator is used for the SOS and WMS servers. You MUST CHANGE these to describe your installation. --> <adminInstitution>NOAA Environmental Research Division</adminInstitution> <adminIndividualName>Your Name</adminIndividualName> <adminPosition>Webmaster</adminPosition> <adminPhone>your-phone-number</adminPhone> <adminAddress>99 Pacific St, Suite 255A</adminAddress> <adminCity>Monterey</adminCity> <adminStateOrProvince>CA</adminStateOrProvince> <adminPostalCode>93940</adminPostalCode> <adminCountry>USA</adminCountry> <adminEmail>yourName@yourSite</adminEmail> <!-- Information about the ERDDAP administrator is used for ERDDAP's SOS server. You MUST CHANGE these to describe your installation. --> <sosTitle>NOAA Environmental Research Division SOS</sosTitle> <sosAbstract>NOAA Environmental Research Division's ERDDAP makes data from multiple sources available via the SOS protocol.</sosAbstract> <sosKeywords>Weather, Ocean Currents, Temperature, Salinity</sosKeywords> <sosAccessConstraints>NONE</sosAccessConstraints> <sosFees>NONE</sosFees> <!-- Information about the ERDDAP administrator is used for ERDDAP's WMS server. You MUST CHANGE these to describe your installation. --> <wmsTitle>NOAA Environmental Research Division WMS</wmsTitle> <wmsAbstract>NOAA Environmental Research Division's ERDDAP makes data from multiple sources available via the WMS protocol.</wmsAbstract> <wmsKeywords>Weather, Ocean Currents, Temperature, Salinity</wmsKeywords> <wmsAccessConstraints>NONE</wmsAccessConstraints> <wmsFees>NONE</wmsFees> <!-- For the wms examples, pick one of your grid datasets that has longitude and latitude axes. The sample variable must be a variable in the sample grid dataset. The bounding box values are minx,miny,maxx,maxy. --> <wmsSampleDatasetID>erdBAssta5day</wmsSampleDatasetID> <wmsSampleVariable>sst</wmsSampleVariable> <wmsSampleBBox>0,-75,180,75</wmsSampleBBox>
Or,
you can join the ERDDAP Google Group / Mailing List by visiting
https://groups.google.com/forum/#!forum/erddap
and clicking on "Apply for membership".
Once you are a member, you can post your question there
or search to see if the question has already been asked and answered.
ERDDAP, Version 2.19
Disclaimers |
Privacy Policy