NOAA   ERDDAP
Easier access to scientific data

Brought to you by NOAA NMFS SWFSC ERD    
 

ERDDAP Changes

Here are the changes associated with each ERDDAP release.

Changes in ERDDAP version 1.74 (released 2016-10-07)

  • New Features (for users):
     
    • Now, when a List of Datasets (All, or from a search) is displayed on a web page, long titles are displayed on multiple lines. Previously, the middle of a long title was replaced by " ... ". Thanks to Margaret O'Brien, LTER, and EML.
       
  • Things ERDDAP Administrators Need to Know and Do:
     
    • TO DO: On Linux computers, change the Apache timeout settings so that time-consuming user requests don't timeout (with what often appears as a "Proxy" or "Bad Gateway" error). As the root user:
      1. Modify the Apache httpd.conf file (usually in /etc/httpd/conf/ ):
        Change the existing <Timeout> setting (or add one at the end of the file) to 3600 (seconds), instead of the default 60 or 120 seconds.
        Change the existing <ProxyTimeout> setting (or add one at the end of the file) to 3600 (seconds), instead of the default 60 or 120 seconds.
      2. Restart Apache: /usr/sbin/apachectl -k graceful (but sometimes it is in a different directory).
      Thanks to Thomas Oliver.
       
    • NEW: [bigParentDirectory/hardFlag directory
      This works like the flag directory, but the hardFlag version also deletes of the all cached dataset information. There are no URLs to set a hardFlag. This can only be used by putting a file in that directory.
      hardFlags are very useful when you do something that causes a change in how ERDDAP reads and interprets the source data, for example, when you install a new version of ERDDAP or when you have made certain types of changes to a dataset's definition in datasets.xml. See this documentation. Thanks to John Kerfoot and all the Argo groups.
       
    • NEW: GenerateDatasetsXml now has a EDDTableFromEML option
      which reads a dataset description in an Ecological Metadata Language (EML) file, downloads the related data file, and generates a chunk of datasets.xml so that the dataset can be added to ERDDAP. There is also an EDDTableFromEMLBatch which does the same thing for all of the EML files in a directory. This works very well because EML does an excellent job of describing the dataset and because KNB and LTER make the actual data files available.
      EML plus ERDDAP could be a great combination, since ERDDAP could give users more direct access to the wealth of KNB and LTER data and help those projects meet the US government's Public Access to Research Results (PARR) requirements by making the data available via a web service.
      See this documentation. Thanks to Margaret O'Brien, LTER, and EML.
       
    • NEW: GenerateDatasetsXml now has a EDDTableFromInPort option
      which reads a dataset description in an InPort XML file and tries to generate a chunk of datasets.xml so that the dataset can be added to ERDDAP. This rarely creates a ready-to-use chunk of XML for datasets.xml, but it will create a good rough draft that is a good starting point for editing by a human.
      It would be great if people using InPort to document their datasets would also use ERDDAP to make the actual data available via ERDDAP's web services and thereby meet the US government's and NOAA's Public Access to Research Results (PARR) requirements by making the data available via a web service. This is a solution that could be used right now. (bob.simons at noaa.gov is happy to help.)
      See this documentation. Thanks to Evan Howell and Melanie Abecassis.
       
    • CHANGE: ERDDAP now uses netcdf-java 4.6.6.
      With earlier versions, netcdf-java read some fill values (perhaps, just in netcdf-4 files) as 0's. Now it reads some of them as the netcdf standard fill value: -127 for bytes, -32767 for shorts, -2147483647 for ints. Unidata says the new behavior is the proper behavior. If a variable in a dataset starts showing one of these values where they used to show 0's, you can add, e.g.,
      <att name="_FillValue" type="short">-32767</att>
      to the variable's addAttributes to tell ERDDAP to treat that value as a missing_value/_FillValue. However, in many cases, that will not yield the desired result: 0's. If so, consider modifying the files with NCO or rewriting the files. Complaints? Please contact Unidata ;-)
       
    • TO DO: New TopographyDepth palette
      I encourage you to switch all datasets that use the OceanDepth palette to use the new TopographyDepth palette, which is like Topography except with the colors flipped, so that it is suitable for depth values (positive=down), instead of altitude values (positive=up). The recommended settings for this palette are:
          <att name="colorBarMaximum" type="double">8000.0</att>
          <att name="colorBarMinimum" type="double">-8000.0</att>
          <att name="colorBarPalette">TopographyDepth</att> 
    • NEW FEATURE: String missing_value and/or _FillValue
      If a String variable defines a missing_value and/or _FillValue, ERDDAP will now remove those values from the data and replace them with an empty string, so that missing values appear as empty strings, as with other datasets in ERDDAP. Thanks to Margaret O'Brien, LTER, and EML.
       
    • NEW FEATURE: Support for Local Times
      timestamp variables with source data from Strings can now specify a time zone via a "time_zone" attribute which leads ERDDAP to convert the local-time-zone source times (some in Standard time, some in Daylight Savings time) into Zulu times. The list of valid time zone names is probably identical to the list in the TZ column in this table. The default is "Zulu". Common US time zones are: US/Hawaii, US/Alaska, US/Pacific, US/Mountain, US/Arizona, US/Central, US/Eastern. For timestamp variables with numeric source data, you can specify the "time_zone" attribute, but the value must be "Zulu" or "UTC". Thanks to Margaret O'Brien, LTER, and EML.
       
    • NEW FEATURE: EDDTableFromAsciiFiles now supports semicolon-separated files
      and is smarter about figuring out the separator. Thanks to Margaret O'Brien, LTER, and EML.
       
    • NEW FEATURE: If there is a significant error in loadDatasets (major or minor, e.g., a missing or invalid datasets.xml document), ERDDAP will now indicate it in status.html, right below "n Datasets Failed To Load" as ERROR: while processing datasets.xml: see log.txt for details.
       
    • NEW FEATURE: ERDDAP looks for orphans.
      When ERDDAP does a major loadDatasets, it now looks for orphan datasets (datasets that are in ERDDAP but not in datasets.xml). If found, they are listed in status.html, right below "n Datasets Failed To Load" as ERROR: n Orphan Datasets (datasets in ERDDAP but not in datasets.xml) = ....
      If you want to remove (unload) an orphan from ERDDAP, you need to add
      <dataset type="anyValidType" datasetID="theDatasetID" active="false" />
      to datasets.xml until the dataset is unloaded during the next major loadDatasets.
       
    • BUG FIX: If a dataset had a numeric timestamp variable with units other than "seconds since 1970-01-01T00:00:00Z" and with the <updateEveryNMillis> system active, the timestamp variable's range was set incorrectly when the dataset was updated. Thanks to John Kerfoot.
       
    • BUG FIX: If <quickRestart> was true in setup.xml and you requested data from an EDDTableFrom...Files dataset that used , the first request to the dataset would fail, but subsequent requests would succeed. Now the first request won't fail. Thanks to John Kerfoot.
       
    • BUG FIX: The GenerateDatasetsXml.sh and .bat didn't work with >9 parameters on the command line. Now they do. Thanks to John Kerfoot.
       
    • BUG FIX: The new EDDTableFromMultidimNcFiles didn't consistently remove trailing spaces from strings. Now it does. Notably, this affected ARGO files. Thanks to Kevin O'Brien and Roland Schweitzer.
       
    • BUG FIX: All access of remote DAP services is now initiated by more modern code. This fixes the "connection closed" error when accessing some EDDTableFromErddap datasets. Thanks to Kevin O'Brien.
       
    • BUG FIX: The handling of orderBy...() and distinct() are now back to the way they were before the recent changes: a given request may have multiple orderBy...() and/or a distinct() filter; ERDDAP will handle them in the order they are specified. Thanks to David Karuga.
       
    • BUG FIX: If the dataset is EDDTableFromDatabase and a query has sourceCanOrderBy and/or sourceCanDoDistinct, then the database may (depending on settings in datasets.xml) partly or completely handle only the first orderBy..() or distinct(). Thanks to David Karuga.
       
    • BUG FIX: The recent extra percent-encoding caused problems with some queries for .ncCF files, e.g., "HTTP Status 500 - Query error: variable=station is listed twice in the results variables list." Thanks to Kevin O'Brien.
       
    • BUG FIX: EDDTableFromFiles had trouble reloading a dataset when one of the columns was a true char column. Thanks to Roland Schweitzer.
       
    • BUG FIX: EDDGridFromNcFilesUnpacked now also converts missing_value and _FillValue to standard values so files with different values can be aggregated. Because of this change, after you install this new version of ERDDAP, please set a hardFlag for each EDDGridFromNcFilesUnpacked dataset in your ERDDAP.
       
    • CHANGE: EDDTableFromNcCFFiles can now handle files that have multiple sample_dimension's. A given dataset must only use variables that use one of the sample_dimensions. Thanks to Ajay Krishnan.
       
    • CHANGE: For EDDTableFrom...Files, <sortFilesBySourceNames> now allows comma-separated (recommended) or space separated lists of variable source names. In either case, individual variable names may be surrounded by double quotes, e.g., if the name has an internal space.

Changes in ERDDAP version 1.72 (released 2016-05-12)

  • New Features (for users): None.
     
  • Things ERDDAP Administrators Need to Know and Do:
    • NEW EDDTableFromMultidimNc EDDTableFromMultidimNc is a new alternative to EDDTableFromNc. It is designed to deal with groups of files with several variables with shared dimensions, e.g., var1[a][b], var2[a], var3[b], scalarVar. Thanks to the Argo Project, Aurélie Briand, and Roland Schweitzer.
    • BUG FIX: ERDDAP (via the FileVisitorDNLS and FileVistorSubdir classes) now follows symbolic links on Linux. ERDDAP still doesn't follow .lnk's on Windows.
    • BUG FIX of bug introduced in 1.70: distinct + orderBy were not allowed together in one request. Now they are again. They are not mutually exclusive/redundant. Thanks to David Karuga.
    • CHANGE to datasets.xml blacklist of IP addresses:
      IP v4 addresses appear to ERDDAP as 4 period-separated hex numbers.
      I think IP v6 addresses appear as 8 colon-separated hex numbers.
      So ERDDAP now supports colons in the IP addresses in that list and :* at the end of the list to block a range of addresses.
    • CHANGE: ERDDAP now uses NetcdfFileWriter to write .nc files instead of the deprecated NetcdfFileWriteable. There should be no discernable change to the resulting files. This opens up the possibility of making big .nc files that use the .nc3 64bit extensions. If you want/need that, please send a request to bob.simons at noaa.gov .
    • CHANGE: Many of the links to remote websites were out-of-date. Now they are up-to-date and use https: instead of http: whenever possible.
    • Many small changes.

Changes in ERDDAP version 1.70 (released 2016-04-15)

  • New Features (for users): None.
     
  • Things ERDDAP Administrators Need to Know and Do: Below, there are several recommended changes to the documentation in your setup.xml file.
    Please make these changes now.
    30 minutes work now may save you hours of confusion in the future.
    • Bug fix: The problem was that requests which were redirected to a remote ERDDAP failed with a invalid character '|' error message. This only occurred with recent versions of Tomcat. Thanks to Rusty Holleman, Conor Delaney, and Roy Mendelssohn.
    • Bug fix: ERDDAP now uses an up-to-date version of netcdf-java (it's a long story) which includes up-to-date support for NcML, which fixes the problem with NcML LogicalReduce not working as expected. There may be a few small changes to the metadata which ERDDAP reads via netcdf-java from .nc, .hdf, .grib, and .bufr files. Thanks to Favio Medrano.
    • The new EDDTableAggregateRows allows you to make a merged EDDTable dataset from two or more EDDTable datasets which have the same data variables using the same units. Thanks To Kevin O'Brien.
    • New options for EDDTableFromDatabase (sourceCanOrderBy and sourceCanDoDistinct) let you specify whether ERDDAP, the database, or both, handle distinct and orderBy (and all variants) constraints. Thanks to David Karuga.
    • You can now make a private dataset's graphs and metadata available to the public via the new <graphsAccessibleTo>public</graphsAccessibleTo> tag. Thanks to Emanuele Lombardi.
    • Now, if a string passed to GenerateDatasetsXml or DasDds is surrounded by double quotes, it is unquoted (as if it is a JSON string). Thanks to John Kerfoot and Melanie Abecassis.
    • GenerateDatasetsXml now supports "default" to get the default and "nothing" to get an empty string (they work with or without quotes). This solves some problems related to passing empty strings.
    • Now, in GenerateDatasetsXml, for all EDDGridFromFiles and EDDTableFromFiles datasets, if the sampleFileName you specify is "" (the empty string), it will use the last matching fileName from the directory + regex + recursive=true.
    • Updated: The displayInBrowser code which is used to display the results of GenerateDatasetsXml and DasDds on Linux computers was out-of-date and gave an odd message about Netscape. Now, this uses a modern Linux tool: xdg-open. Thanks to Melanie Abecassis.
    • The allDatasets dataset now has a "files" column, which indicates the base URL of the /files link (if there is one) for the dataset.
    • Increase the general security of your ERDDAP by changing the permissions associated with the tomcat directory and the bigParentDirectory:
      (The actual commands below are for Linux. For other OS's, make analogous changes.)
      • Change the "group" to be tomcat, your username, or the name of a small group that includes tomcat and all the administrators of Tomcat/ERDDAP, e.g.,
        chgrp -R yourUserName apache-tomcat-8.0.23
        chgrp -R yourUserName bigParentDirectory
      • Change permissions so that tomcat and the group have read, write, execute privileges, e.g,.
        chmod -R ug+rwx apache-tomcat-8.0.23
        chmod -R ug+rwx bigParentDirectory
      • Remove "other" user's permissions to read, write, or execute:
        chmod -R o-rwx apache-tomcat-8.0.23
        chmod -R o-rwx bigParentDirectory
        This is important, because it prevents other users from reading possibly sensitive information in ERDDAP setup files, log files, and files with information about private datasets.
    • The authentication/login system was revamped. Thanks to Thomas Gardner, Emanuele Lombardi, and the U.S. government's new HTTPS-Only Standard.
      • The authentication=openid option was removed. It was out-of-date.
      • The new, recommended, authentication=google option uses Google Sign-In (based on OAuth 2.0) to allow anyone with a Google email account (including Google managed accounts like @noaa.gov) to log in.
      • The new, authentication=email option is a back up for authentication=google. It allows users with a <user> tag in datasets.xml to log in by sending them an email with a special link.
      • In your setup.xml, please change the description for <authentication> to be
        <!-- If you want to restrict access to some datasets, 
        you need to specify the method used for logging on (authentication).
        See the info at 
        https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security
        Currently, the options are: "" (logins not supported, the default), 
        "custom", "email", and "google" (recommended).  
        [No longer supported: "basic", "openid"]
        -->
      • In your setup.xml, please add this right below the <authentication> tag
        <!-- If authentication=google, you must supply your Google Client ID. 
        See
        https://developers.google.com/identity/sign-in/web/devconsole-project
        When setting this up, for Authorized JavaScript origins, 
        for testing on your computer, use the domain "localhost" 
        (e.g., origin=https://localhost:8443), 
        not "127.0.0.1" (because Google Sign-In doesn't work with anything 
        at that domain).
        This will be a string of about 75 characters, probably starting with
        several digits and ending with .apps.googleusercontent.com .
        -->
        <googleClientID></googleClientID>
      • Now, users who aren't logged in can use http or https URLs (if you have set up <baseHttpsUrl> in your setup.xml). Thanks to the U.S. government's new HTTPS-Only Standard.
      • Now, you can encourage all users to use https (not http) by setting <baseUrl> to be an https URL. To force users to use only https, you must also make changes to your Apache/Tomcat setup to block non-https access. Thanks to the U.S. government's new HTTPS-Only Standard.

        In your setup.xml, please change the description for <baseUrl> to be

        <!-- baseUrl is the start of the public url, to which "/erddap" 
        is appended. For example:
        For running/testing on your personal computer:
          <baseUrl>http://localhost:8080</baseUrl>     
          (127.0.0.1 doesn't work with authentication=google).
        For ERD releases, we use
          <baseUrl>http://coastwatch.pfeg.noaa.gov</baseUrl>    
        If you want to encourage all users to use https (not http), 
          make the baseUrl the same as the baseHttpsUrl (see below).
        -->
      • The options <passwordEncoding> changed. In your setup.xml, please change the description for <passwordEncoding> to be
        <!-- For "custom" authentication, this specifies how you have 
        stored passwords in the roles tags in datasets.xml.
        If you aren't storing any passwords, this is irrelevant.
        The options (in order of increasing security) are: 
        "MD5", "UEPMD5" (MD5(UserName:ERDDAP:Password)), 
        "SHA256", "UEPSHA256" (SHA256(UserName:ERDDAP:Password), 
        the default).
        You should only use "MD5" or "SHA256" if you need to match 
        values stored that way in an external password database.
        See the info at 
        https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security
        --> 
      • In your setup.xml, please change the description for <baseHttpsUrl> to be
        <!-- This is a variant of <baseUrl> which is used when 
        authentication is active and the user is logged in.
        In general, you take the <baseUrl>, change "http" to "https", 
        and change/add ":8443". This must begin with "https://".
        If you make a proxy so that ":8443" isn't needed, 
        then don't use ":8443" here.
        This is relevant even if <authentication> is "".
        See the instructions at 
        https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security
        For example:
        For running/testing on your personal computer:
          <baseHttpsUrl>https://localhost:8443</baseHttpsUrl>                  
        For releases at ERD, we use:
          <baseHttpsUrl>https://coastwatch.pfeg.noaa.gov</baseHttpsUrl>  
        If you want to encourage all users to use https (not http), 
          make the baseUrl (see above) the same as the baseHttpsUrl.
        --> 
      • Now, if listPrivateDatasets=true in setup.xml, even less information will be shown about datasets that a user doesn't have access to.
    • Now, especially for when you are initially setting up your ERDDAP, you can now tell ERDDAP not to try to subscribe to remote ERDDAP datasets. Thanks to Filipe Rocha Freire.
      In your setup.xml, right before <fontFamily>, please add
      <!-- Normally, if you have a EDDGridFromErddap or EDDTableFromErddap 
      dataset in your datasets.xml, it will try to subscribe to the remote 
      ERDDAP dataset so that the local dataset is kept perfectly up-to-date.
      If this ERDDAP is not publicly accessible (http://localhost), or its
      IP address will change soon, or you have some other reason, 
      you can tell this ERDDAP to not try to subscribe to the remote 
      ERDDAP datasets by setting this to false. (default=true) 
      This is the overall setting for this ERDDAP. It can be overridden by
      the same tag (with a different value) in the datasets.xml chunk for 
      a given EDD...FromErddap dataset. 
      For each fromErddap dataset that doesn't subscribe to the remote 
      ERDDAP dataset, you should set <reloadEveryNMinutes> to a smaller 
      number so that the local dataset stays reasonably up-to-date. -->
      <subscribeToRemoteErddapDataset>true</subscribeToRemoteErddapDataset>
    • In your setup.xml, in the instructions above <emailFromAddress>, please insert:
      If possible, set this up to use a secure connection (SSL / TLS) to the email server.
      If your setup isn't using a secure connection to the email server, please make the changes to make it so.
    • In your datasets.xml, please add this line to the description of <subscriptionEmailBlacklist> in your datasets.xml:
      You can use the name "*" to blacklist an entire domain, e.g., *@example.com .
    • Since the change to the logging system in v1.66, the log file is never up-to-date. There are always messages or parts of messages waiting to be written to the log file. Now, you can make it up-to-date (for an instant) by viewing your ERDDAP's status web page at http://your.domain.org/erddap/status.html .
    • HashDigest .......
    • A small change (to String2.canonical) that should help keep things moving quickly when ERDDAP is very busy and also better deal with a very large number of datasets.
    • Strongly Recommended: stop using <convertToPublicSourceUrl> in datasets.xml to convert an IP number in a dataset's <sourceUrl> (e.g., http://192.168.#.#/) into a domain names (e.g., http:my.domain.org/). From now on, new subscriptions to http://localhost, http://127.0.0.1, and http://192.168.#.# URLS won't be allowed for security reasons. So please always use the public domain name in the <sourceUrl> tag (if needed because of DNS problems), you can use the /etc/hosts table on your server to solve the problem by converting local domain names to IP numbers without using a DNS server. You can test if a given domain name gets properly resolved by using
      ping some.domain.name
    • In generateDatasets.xml, for remote datasets (e.g., from a THREDDS server), the automatically generated datasetIDs are unchanged for most domains. For a few domains, the first part (i.e., the name) of the automatically generated datasetID will be a little different. Notably, names that had one part are now more likely to have two parts. For example, datasets from http://oos.soest.hawaii.edu previously led to datasetIDs that started with hawaii_, but now lead to datasetIDs that start with hawaii_soest_ . If this causes problems for you, please email me. There may be a work-around.
    • The Cassandra driver was updated to cassandra-driver-core-3.0.0.jar and thus for Cassandra v3. EDDTableFromCassandra doesn't take advantage of any new features in Cassandra v3. Indexes in Cassandra can now be more complex, but ERDDAP still uses the Cassandra v2 index model, which assumes that an indexed column can be directly queried with '=' constraints. GenerateDatasetsXml for EDDTableFromCassandra no longer detects columns with indexes; if an index is simple, you need to specify it in datasets.xml by hand. If you need support for more complex indexes or other new features, please email bob.simons at noaa.gov .
      !!! If you still use Cassandra 2.x, please continue to use ERDDAP v1.68 until you upgrade to using Cassandra 3.x.
    • Jars and the Classpath - Almost all of the included third-party .jar files were updated to their latest versions.
      • slf4j.jar was added to /lib and the classpath.
      • joid.jar and tsik.jar were removed from /lib and the classpath.
      • If you get error messages about classes not found when you compile or run ERDDAP or one of its tools, compare your command line's classpath to ERDDAP's current classpath to figure out which .jars are missing from your classpath.

Changes in ERDDAP version 1.68 (released 2016-02-08)

  • New Features (for users): None.
     
  • Things ERDDAP Administrators Need to Know and Do:
    • EDDGridFromFiles Aggregation via File Names or Global Metadata -
      All variations of EDDGridFromFiles can now aggregate a group of files by adding a new leftmost dimension, usually time, based on a value derived from each file name or from the value of a global attribute that is in each file.
    • Change: We previously suggested that you might like to create an EDDGridFromErddap dataset in your datasets.xml that referenced and re-served the jplMURSST dataset in our ERDDAP. Since there is now a newer version of that dataset, that dataset is now deprecated. So if you have that dataset in your ERDDAP, please add this new dataset
      <dataset type="EDDGridFromErddap" datasetID="jplMURSST41" active="true">
        <!-- Multi-scale Ultra-high Resolution (MUR) SST analysis fv04.1, Global, 0.011 Degree, Daily -->
        <sourceUrl>https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41</sourceUrl>
      </dataset>

      If you want to remove the old jplMURSST dataset from your ERDDAP (it's your choice), change its active setting from "true" to "false".
    • Bug fix: Please check the bigParentDirectory that you specified in your setup.xml. If you didn't put a slash at the end of the <bigParentDirectory> name, then ERDDAP will have created several directories by appending words directly to the name that you specified, instead of creating subdirectories. Starting with version 1.68, ERDDAP adds a slash to the end of the directory name if you didn't specify one. So if you previously didn't specify a slash at the end, then when you install ERDDAP v1.68 you need to move and rename those directories after you shutdown the old ERDDAP and before you startup the new ERDDAP. For example, if you mistakenly specified bigParentDirectory as /home/erddapBPD (no trailing slash) and ERDDAP has mistakenly created directories like
      /home/erddapBPDcache
      /home/erddapBPDcopy
      /home/erddapBPDdataset
      /home/erddapBPDflag
      /home/erddapBPDlogs
      /home/erddapBPDlucene
      and a file named /home/erddapBPDsubscriptionsV1.txt,
      then you need to move and rename them to be
      /home/erddapBPD/cache
      /home/erddapBPD/copy
      /home/erddapBPD/dataset
      /home/erddapBPD/flag
      /home/erddapBPD/logs
      /home/erddapBPD/lucene
      and /home/erddapBPD/subscriptionsV1.txt
    • Bug fix: There were bugs in EDDGridLonPM180 in ERDDAP v1.66 that occurred when the child dataset is an EDDGridFromErddap.
    • Bug fix: There was a bug in EDDGridFromFiles and EDDTableFromFiles in ERDDAP v1.66 that caused <updateEveryNMillis> to be ignored the first time the dataset was loaded after a restart.
    • Bug fix/New Feature: If a child dataset within EDDGridAggregateExistingDimension, EDDGridCopy, EDDGridFromEDDTable, EDDGridLonPM180, EDDGridSideBySide, EDDTableCopy, or EDDTableFromEDDGrid is a ...FromErddap dataset, that parent dataset now subscribes to the underlying ERDDAP dataset. If the underlying ERDDAP dataset is in the same ERDDAP, the subscription and its validation are done directly; you won't get an email asking you to validate the subscription. Otherwise, if the subscription system for your ERDDAP is turned off, set the <reloadEveryNMinutes> setting for the parent dataset to a smallish number (60?) so that it stays up-to-date.
    • Bug fix/New Feature: If a child dataset within EDDGridAggregateExistingDimension, EDDGridCopy, EDDGridFromEDDTable, EDDGridLonPM180, EDDGridSideBySide, EDDTableCopy, or EDDTableFromEDDGrid has active="false", that child dataset is now skipped.

Changes in ERDDAP version 1.66 (released 2016-01-19)

  • New Features (for users):
    • Graphs (not maps) can now have descending values on the axes. To get this when using a Make A Graph web page, change new Y Axis : ascending setting (the default) to descending. Or, in a URL that requests a graph, use the new optional 3rd '|' parameter for the &.xRange and/or &.yRange switches, which can be nothing (the default), true, or t to get ascending values, or use false or f to get descending values. The true|false values are case insensitive. Thanks to Chris Fullilove, John Kerfoot, Luke Campbell, and Cara Wilson.
    • Users can now specify the background color for graphs by adding a &.bgColor=0xAARRGGBB switch to the URL which requests the graph. See .bgColor in the Graphics Commands section of the griddap and tabledap documentation. Thanks to John Kerfoot and Luke Campbell.
    • For tabular datasets, constraints can now refer to min(someVariableName) or max(someVariableName) . See min() and max(). Thanks to John Kerfoot.
    • For tabular datasets, time constraints that use now can now specify time units of milliseconds or millis.
    • A request for an image of a tabular dataset now makes a map (not a graph) if the x and y variables are longitude-like and latitude-like variables (compatible units). Thanks to Rich Signell.
    • Bug fix: Time axis labels and ticks sometimes had odd irregularities when requesting multiple graphs simultaneously (e.g., on a web page). The problem was a bug in the SGT graphics library that ERDDAP uses (one variable was "static" that shouldn't have been). Thanks to Bradford Butman.
       
  • Things ERDDAP Administrators Need to Know and Do:
    • It is a security risk to put your email password in a plain text file like setup.xml. To mitigate that problem, we strongly recommend that you:
      1. Set up an email account just for ERDDAP's use, e.g., erddap@yourInstitution.org . That has other benefits as well; notably, more than one ERDDAP administrator can then be given access to that email account.
      2. Make the permissions of the setup.xml file rw (read+write) for the user who will run Tomcat and ERDDAP (user=tomcat?) and no permissions (not read or write) for the group and other users.
      Thanks to Filipe Rocha Freire.
    • The new ArchiveADataset tool simplifies making a .tar.gz archive with a subset of a dataset in a format that is suitable for archiving (notably, at NOAA's NCEI). This should be useful for many ERDDAP administrators in many situations, but especially for groups within NOAA.
    • The new dataset type EDDGridFromNcFilesUnpacked is a variant of EDDGridFromNcFiles. The difference is that this class unpacks each data file before EDDGridFromFiles looks at the files:
      • It unpacks packed variables that use scale_factor and/or add_offset.
      • It promotes integer variables that have _Unsigned=true attributes to a larger integer data type so that the values appear as the unsigned values. For example, an _Unsigned=true byte (8 bit) variable becomes a signed short (16 bit) variable.
      • It converts _FillValue and missing_value values to be NaN's (or MAX_VALUE for integer data types).
      The big advantage of this class is that it provides a way to deal with different values of scale_factor, add_offset, _FillValue, or missing_value in different files in a collection. Otherwise, you would have to use a tool like NcML or NCO to modify each file to remove the differences so that the files could be handled by EDDGridFromNcFiles. For this class to work properly, the files must follow the CF standards for the related attributes. Thanks to Philippe Makowski.
    • The new dataset type EDDGridLonPM180 lets you change datasets that have some longitude values greater than 180 (e.g., the range 0 to 360) into datasets with longitude values within the range -180 to 180 (Longitude Plus or Minus 180, hence the name). The big advantage to offering datasets with longitude values in the range -180 to 180 is that OGC services (e.g., WMS) require longitude values in this range. Thanks to Lynne Tablewski, Fabien Guichard, Philippe Makowski, and Martin Spel.
      2016-01-26 Update: Eeek! This has a bug that occurs when the child dataset is an EDDGridFromErddap that references a dataset in the same ERDDAP. This bug is fixed in ERDDAP v1.68.
    • In GenerateDatasetsXml, a new special dataset type, EDDGridLonPM180FromErddapCatalog, lets you generate the datasets.xml for EDDGridLonPM180 datasets from all of the EDDGrid datasets in an ERDDAP that have any longitude values greater than 180.
    • For all EDDGrid datasets, in datasets.xml you can now use the optional
      <accessibleViaWMS>true|false</accessibleViaWMS> (default=true). Setting this to false forcibly disables the WMS service for this dataset. If true, the dataset may still not be accessible via WMS for other reasons (e.g., no lat or lon axes). This is particularly useful for datasets that exist on their own and wrapped by EDDGridLonPM180, so that only the LonPM180 version is accessible via WMS.
    • In setup.xml, you can specify a different default color for the background of graphs. The color is specified as a 8 digit hexadecimal value in the form 0xAARRGGBB, where AA, RR, GG, and BB are the opacity, red, green and blue components, respectively, specified as 2-digit hexadecimal numbers. Note that the canvas is always opaque white, so a (semi-)transparent graph background color blends into the white canvas. The default is light blue:
      <graphBackgroundColor>0xffccccff</graphBackgroundColor>
      Thanks to John Kerfoot and Luke Campbell.
    • In setup.xml, you can now specify the maximum size for the log file (when it is renamed to log.txt.previous and a new log.txt is created), in MegaBytes. The minimum allowed is 1. The maximum allowed is 2000. The default is 20 (MB). For example:
      <logMaxSizeMB>20</logMaxSizeMB>
    • In datasets.xml, <fgdcFile> or <iso19115File> can now be a local file (as before) or a URL (which will be downloaded so there is a local copy). If ERDDAP is unable to download the file, the loading of the dataset will continue but the dataset won't have an fgdc or iso19115 file.
    • EDDGridFromFiles and EDDTableFromFiles datasets can now do a quickRestart (the system that ERDDAP tries to use when datasets are first loaded when ERDDAP is restarted). This speeds up restarting ERDDAP.
      2016-01-26 Update: Eeek! This has a bug that causes <updateEveryNMillis> to be ignored the first time the dataset is loaded after a restart. This bug is fixed in ERDDAP v1.68.
    • A general improvement to the quickRestart system allows ERDDAP to load datasets faster when ERDDAP is restarted.
    • All EDDGridFromFiles and EDDTableFromFiles subclasses now accept a new <pathRegex> tag, usually specified right below <recursive>. If recursive is "true", only full subdirectory paths which match the pathRegex (default=".*") will be accepted. Similarly, a <sourceUrls> tag in an EDDGridAggregateExistingDimension can now include a pathRegex attribute (default=".*").
    • The default for <partialRequestMaxBytes> in setup.xml is now 490000000 (~490 MB). This avoids some problems/timeouts related to getting data from THREDDS data servers. Thanks to Leslie Thorne.
    • A small change to the log system should allow ERDDAP to be more responsive when it is very, very busy. Information is now written to the log file on the disk drive in fairly big chunks. The advantage is that this is very efficient -- ERDDAP will never block waiting for information to be written to the log file. The disadvantage is that the log will almost always end with a partial message, which won't be completed until the next chunk is written.
    • Bug fix related to inotify and the <updateEveryNMillis> system for EDDGridFromFiles and EDDTableFromFiles datasets: It is no longer necessary to specify a large of fs.inotify.max_user_watches or fs.inotify.max_user_instances. There is a bug in Java that causes some parts of Java's inotify/WatchDirectory system to be not garbage collected when they are finalized; eventually, the number of zombie inotify watches or instances would exceed the maximum number specified. ERDDAP now works around this Java bug.
      Also, the number of inotify threads is listed on the status.html web page, so you can keep an eye on its usage. Typically, there is 1 inotify thread per EDDGridFromFiles and EDDTableFromFiles dataset.
    • Bug fix: in many places, instead of an error being rethrown, a new error was generated which only included a short version of the original error message and without the stack trace. Now, when a new error is generated, it properly includes the entire original exception e.g., throw new Exception("some new message", e);
      Thanks to Susan Perkins.
    • Bug fix: until recently (v1.64?), if a .../datasetID URL was requested, ERDDAP would add .html to the URL. In v1.64, this failed (an incorrectly formatted URL was generated and then failed). Now this works again. Thanks to Chris Fullilove.

Changes in ERDDAP version 1.64 (released 2015-08-19)

  • New Features (for users):
    • There is now guidance for accessing the password-protected private ERDDAP datasets (https://) via curl and Python. See the curl and Python instructions.
      Thanks to Emilio Mayorga of NANOOS and Paul Janecek of Spyglass Technologies.
       
  • Things ERDDAP Administrators Need to Know and Do:
    • ERDDAP now requires Java 1.8+.
      Java 1.7 reached its end of life (external link) (no more security updates) in April 2015. This version of ERDDAP will not work with versions of Java below 1.8. If you update from Java 1.7x (or earlier), you should also update Tomcat. See the ERDDAP Set Up Instructions for download links and advice.
    • New Data Provider Form.
      When a data provider comes to you hoping to add some data to your ERDDAP, it can be difficult and time consuming to collect all of the metadata needed to add the dataset into ERDDAP. Many data sources (for example, .csv files, Excel files, databases) have no internal metadata. So ERDDAP has a new Data Provider Form which gathers metadata from the data provider and gives the data provider some other guidance, including extensive guidance for Data In Databases. The information submitted is converted into the datasets.xml format and then emailed to the ERDDAP administrator (you) and written (appended) to bigParentDirectory/logs/dataProviderForm.log . Thus, the form semi-automates the process of getting a dataset into ERDDAP, but the ERDDAP administrator still has to complete the datasets.xml chunk and deal with getting the data file(s) from the provider or connecting to the database. For more information, see the Data Provider Form description.
    • New <matchAxisNDigits>
      can be used by EDDGridFromFiles (and thus fromNcFiles and fromMergeIRFiles), EDDGridAggregateExistingDimension, EDDGridCopy, and EDDGridSideBySide datasets to specify how precisely equal the axis values in different files must be (how many digits): 0=no checking (don't use this!), 1-18 for increasing precision, or 20 (the default) for exact equality. For n=1-18, ERDDAP ensures that the first n digits of double values (or (n+1) div 2 for float values) are equal.
      <matchAxisNDigits> replaces <ensureAxisValuesAreEqual>, which is now deprecated. A value of 'true' will be converted to matchAxisNDigits=20. A value of 'false' (don't do this!) will be converted to matchAxisNDigits=0.
    • EDDGridFromFiles and EDDTableFromFiles will load very slowly the first time you use this version of ERDDAP.
      ERDDAP now stores the internal file information a little differently, so the internal file table for each of these datasets has to be rebuilt. So don't worry. Nothing is wrong. It's a one time thing.
    • Remote Source Files
      EDDGridFromNcFiles, EDDTableFromNcFiles, EDDTableFromNcCFFiles now allow the files to be remote files in a directory accessible by http:// (and probably https:// and ftp://, but they are untested) if the remote server supports Range Requests (external link) in the request header. THREDDS and Amazon S3 support Range Requests, Hyrax does not. This system allows you to access data in remote files without downloading the files (which is helpful if the remote files are too voluminous), but access to these files will be far slower than access to local files or even to a remote OPeNDAP source.
      This includes "files" in an Amazon S3 bucket since they are accessible via http://. If the S3 object names are like file names (with internal /'s like a Linux directory tree), ERDDAP can also make the files accessible via ERDDAP's "files" system. For this to work, your S3 credentials must be in ~/.aws/credentials (on Linux, OS X, or Unix), or C:\Users\USERNAME\.aws\credentials (on Windows) on the server with ERDDAP. See the Amazon SDK documentation (external link).
    • GenerateDatasetsXml has a new, unusual option: EDDsFromFiles.
      This will go through a file system (even a remote system like an Amazon S3 if the objects have file-like names) and create the datasets.xml chunks for a series of datasets. Your mileage may vary. This works well if the files are organized so that all the data files in a given directory (and its subdirectories) are suitable for one dataset (e.g., all SST 1-day composites). Otherwise (e.g., if a diretory contains some SST files and some Chlorophyll-a files), this works poorly but may still be useful.
    • Programmers: new /lib .jar files.
      If you compile ERDDAP, please note the new .jar files in the classpath -cp parameter listed in the ERDDAP Programmer's Guide.
    • sea_water_practical_salinity
      If you use the CF standard name sea_water_salinity for any variable, I encourage you to switch to sea_water_practical_salinity which is available in version 29 of the CF Standard Name Table (and some previous versions -- I didn't know that). This name indicates that this is indeed a Practical Salinity value using Practical Salinity Units (PSU), as opposed to an older g/kg value. The canonical units are different, but still incredibly unhelpful: 1 (presumably implying PSU/PSS-78), as opposed to 1e-3 (presumably implying g/kg) for sea_water_salinity. [Hey, Unidata and CF: We identify values that use other scales, for example Fahrenheit or Celsius, via a units string that is the name of the scale or some variation. Why can't we identify salinity units via their scale, e.g., PSS-78? I know: PSS-78 values are "unitless", but there is an implied scale, isn't there? If I invent a new practical salinity scale where the values are 0.875 times the PSS-78 values, should the canonical units still be "1"? How could a user tell them apart? Units of 1e-3 and 1 are neither descriptive nor helpful to users who are trying to figure out what the numbers indicate.]

Changes in ERDDAP version 1.62 (released 2015-06-08)

  • New Features (for users):
    • For EDDGrid datasets, users can now make Graph Type: Surface graphs with any combination of numeric axes, not just longitude vs. latitude. This lets you make x vs y (projected) graphs and various Hovmöller Diagrams (external link), for example, plotting longitude vs depth, or time vs depth. [Note: if depth is on the Y Axis, it will probably be flipped from what you want. Sorry, un-flipping it is not yet an option.] Thanks to Cara Wilson and Lynn DeWitt.
    • There is a new Oceanic/Atmospheric Acronym Converter which lets you convert a common oceanic/atmospheric acronym to/from a full name.
    • There is a new Oceanic/Atmospheric Variable Names Converter which lets you convert a common oceanic/atmospheric variable name to/from a full name.
  • Things ERDDAP Administrators Need to Know and Do:
    • Java 7/8
      Oracle no longer supports (provides security bug fixes for) Java 7. ERDDAP still supports Java 7, but please move to Java 8. The next release of ERDDAP will probably require Java 8.
    • valid_min/max/range
      Previously and now, if a dataVariable had scale_factor and add_offset metadata, ERDDAP unpacks the data values and removes that metadata. Previously, ERDDAP did not modify/unpack any valid_range, valid_min, valid_max metadata (which usually/should contained packed values) by scale_factor and add_offset. Now it does. Please search your ERDDAP for "valid_" and make sure that all of the variables that have valid_range, valid_min, or valid_max have the correct values when the datasets appear in the new version of ERDDAP. See valid_range/min/max documentation.
    • ACDD-1.3
      Previously, ERDDAP (notably GenerateDatasetsXml) used/recommended the original (1.0) version of the NetCDF Attribute Convention for Dataset Discovery which was referred to as "Unidata Dataset Discovery v1.0" in the global Conventions and Metadata_Conventions attributes. Now, we recommend ACDD version 1.3 which was ratified in early 2015 and is referred to as "ACDD-1.3". Fortunately, ACDD-1.3 is highly backward compatibly with version 1.0. We RECOMMEND that you switch to ACDD-1.3. It isn't hard.
    • GenerateDatasetsXml Attributes
      There were a large number of changes to improve the <addAttributes> values suggested by GenerateDatasetsXml for the global Conventions, creator_name/email/url, keywords, summary, and title attributes and for the variable long_name attribute. Some changes are related to the new use of ACDD-1.3.
    • EDDTableFromSOS datasets
      With the occasional addition of new types of SOS servers and changes to the old servers, it is getting harder for ERDDAP to automatically detect the server type from the server's responses. The use of <sosServerType> (with a value of IOOS_NDBC, IOOS_NOS, OOSTethys, or WHOI) is now STRONGLY RECOMMENDED. If any of your datasets of this type have problems in the new version of ERDDAP, try re-running GenerateDatasetsXml for the SOS server to generate a new chunk of datasets.xml for that dataset. GenerateDatasetsXml will let you try out the different <sosServerType> options until you find the right one for a given server. If you still have problems, please let me know the problem you see and the URL of the server and I will try to help.
    • EDDTableFromFileNames datasets
      Some attributes that were recommended addAttributes are now sourceAttributes. You probably don't have to change anything for existing datasets in your datasets.xml.
    • Bug fix related to certain requests to EDDTableFromNcCFFiles datasets.
      I also added a large number of unit tests to the existing large number of units tests of the underlying methods (there are 100's of scenarios). Thanks to Eli Hunter.
    • Bug fix/small changes to EDDGridFromMergeIR.
      Thanks to Jonathan Lafite and Philippe Makowski
    • Bug fix: EDDGridFromErddap now works even if a remote dataset doesn't have ioos_category variable attributes.
      Thanks to Kevin O'Brien.
    • Bug fix in .graph web page for EDDGrid datasets when there is only one axis variable with more than one value.
      Thanks to Charles Carleton.
    • There were other small improvements, changes, and bug fixes.

Changes in ERDDAP version 1.60 (released 2015-03-12)

  • New Features (for users): none
  • Things ERDDAP Administrators Need to Know and Do:
    • STRONGLY RECOMMENDED: Update your server's robots.txt file to include:
      Disallow: /erddap/files/
    • INotify Problem and Solution:
      On Linux computers, if you are using <updateEveryNMillis> with datasets with type=EDDGridFromFiles, EDDTableFromFiles, EDDGridCopy, EDDTableCopy, or their subclasses, you may see a problem where a dataset fails to load (occasionally or consistently) with the error message: "IOException: User limit of inotify instances reached or too many open files". If so, you can fix this problem by calling (as root):
      echo fs.inotify.max_user_watches=65536 | tee -a /etc/sysctl.conf
      echo fs.inotify.max_user_instances=1024 | tee -a /etc/sysctl.conf
      sysctl -p

      Or, use higher numbers if the problem persists. The default for watches is 8192. The default for instances is 128. [UPDATE: There is a bug in Java which causes inotify instances to be not garbage collected. This problem is avoided in ERDDAP v1.66 and higher. So the better solution is to switch to the latest version of ERDDAP.]
    • NoSuchFileException Bug Fix:
      There was a bug that could cause datasets of type=EDDGridFromFiles, EDDTableFromFiles, EDDGridCopy, EDDTableCopy, or their subclasses to not load occasionally with the error "NoSuchFileException: someFileName". The bug is related to uses of FileVisitor and was introduced in ERDDAP v1.56. The problem is rare and is most likely to affect datasets with a large number of frequently changing data files.
    • There were some small improvements, changes, and bug fixes.

Changes in ERDDAP version 1.58 (released 2015-02-25)

  • New Features (for users):
    • The new "files" system lets you browse a virtual file system and download source data files from many ERDDAP datasets. The "files" system is active by default, but ERDDAP administrators can disable it by putting
      <filesActive>false</filesActive>
      in the ERDDAP setup.xml file. Special thanks to Philippe Makowski, who persisted when I was slow to appreciate the beauty of this idea.
    • time destinationMax - Previously, the time variable of EDDTable datasets with near real time data had a destinationMax of NaN, which implied that the maximum time value for the dataset is recent, but not precisely known and changing frequently. Now, the destinationMax has a real value, indicating the currently-known last time. Many datasets, have continuously updated data. ERDDAP supports accessing the latest data, even if it is after the currently-known last time. Note that the new <updateEveryNMillis> support in EDDGridFromFiles and EDDTableFromFiles datasets updates the time variable's destinationMax. Another consequence of this change is that the datasetID=allDatasets dataset now includes the currently-known last time in the maxTime columns. Thanks to John Kerfoot.
  • Things ERDDAP Administrators Need to Know and Do:
    • STRONGLY RECOMMENDED: Update your server's robots.txt file to include:
      Disallow: /files/
      Disallow: /erddap/files/
    • Sample datasets.xml - Last year, we recommended several excellent datasets in the coastwatch ERDDAP that you could add to your ERDDAP just by adding a few lines to your datasets.xml. If you added the erdVH datasets, please switch to the newer erdVH2 datasets:
      • Make a copy of all the erdVH datasets and change the copied datasetID's from erdVH... to erdVH2... and change the referenced sourceUrl from erdVH... to erdVH2....
      • Set the erdVH... datasets to active="false".
    • All EDDGridFromFiles and EDDTableFromFiles subclasses now support <accessibleViaFiles> to make the source data files accessible via the "files" systems. By default, this system is off for each dataset. You need to add the tag to enable it. Thanks to Philippe Makowski.
    • All EDDGridFromFiles and EDDTableFromFiles subclasses now support <updateEveryNMillis>. By default, this system is off for each dataset. You need to add the tag to enable it. Thanks to Dominic Fuller-Rowell and NGDC.
    • The new EDDTableFromFileNames creates a dataset from information about a group of files in the server's file system, but it doesn't serve data from within the files. For example, this is useful for distributing collections of image files, audio files, video files, word-processing files, and spreadsheet files. This works hand-in-hand with the new "files" system, so that users can download the files. Special thanks to Philippe Makowski, who persisted when I was slow to appreciate the beauty of this idea.
    • The new EDDGridFromEDDTable lets you convert a tabular dataset into a gridded dataset. Thanks to Ocean Networks Canada.
    • The new EDDGridFromMergeIRFiles aggregates data from a group of local MergeIR .gz files. EDDGridFromMergeIRFiles has the distinction of being the first chunk of code contributed to ERDDAP. It was done entirely without our help. Three cheers and special thanks to Jonathan Lafite and Philippe Makowski of R.Tech Engineering.
    • There is a new, optional setup.xml tag, <unitTestDataDir>, which specifies the directory with the unit test data files which are available via a new GitHub repository: https://github.com/BobSimons/erddapTest . For example:
      <unitTestDataDir>/erddapTest/</unitTestDataDir>
      This isn't useful yet, but is part of the move toward making as many of the unit tests runnable by other people as possible. Thanks to Terry Rankine.
    • There were many small improvements, changes, and bug fixes.

Changes in ERDDAP version 1.56 (released 2014-12-16)

  • New Features (for users): (None)
  • Things ERDDAP Administrators Need to Know and Do:
    • You probably already know about EDDGridFromErddap and EDDTableFromErddap which let you link to datasets in other ERDDAPs and have them appear in your ERDDAP. User requests for actual data from these datasets get routed invisibly to the source ERDDAP, so the data doesn't flow through your system or use your bandwidth. There is now a large list of recommended datasets in the sample datasets.xml in erddapContent.zip. To include them in your ERDDAP, all you have do is copy and paste the ones you want into your datasets.xml. Thanks to Conor Delaney.
    • If you compile ERDDAP, you need to add some new .jar files to your classpath -cp switch for javac and java.
    • The new EDDTableFromCassandra handles getting data from Cassandra (external link). Thanks to Ocean Networks Canada.
    • The new EDDTableFromColumnarAsciiFiles handles getting data from ASCII data files with fixed-width columns. Thanks to Philippe Makowski.
    • All EDDGridFromFiles and EDDTableFromFiles subclasses now use a new method, FileVisitor (added to Java in 1.7) to gather information about the files. This may have no benefit for the first gathering of file information for a given dataset but seems to have a huge benefit for subsequent gatherings if done soon, while the OS still has the information cached. Thanks to NGDC.

      We still recommend: If a dataset has a large number of files (e.g., >1,000), the operating system (and thus EDDGridFromFiles and EDDTableFromFiles) will operate much more efficiently if you store the files in a series of subdirectories (one per year, or one per month for datasets with very frequent files), so that there are never a huge number of files in a given directory.

    • Several small improvements to EDDTableFromAsciiFiles.
    • Some improvements to EDDTableFromAsciiServiceNOS, notably to get some additional columns of information from the source. Thanks to Lynn DeWitt.
    • Some small bug fixes related to the ISO 19115 that ERDDAP generates. Thanks to Anna Milan.

Changes in ERDDAP version 1.54 (released 2014-10-24)

  • New Features (for users):
    • Some variables now work with time at the milliseconds precision, e.g., 2014-10-24T16:41:22.485Z. Thanks to Dominic Fuller-Rowell.
  • Small changes/Bug Fixes:
    • Bug fix: with a certain combination of circumstances, EDDGridFromNcFile datasets returned data at reduced precision (e.g., floats instead of doubles). This could only affect data values with > 8 significant figures. My apologies. (And it was a classic computer programming bug: one wrong character.) Thanks to Dominic Fuller-Rowell.
    • Many small changes.
  • Things ERDDAP Administrators Need to Know and Do:
    • Griddap datasets now support timestamp axis variables and data variables (i.e., variables with time values, but a destinationName other than "time"). Thanks to Dominic Fuller-Rowell.
    • ERDDAP now correctly supports milliseconds time_precision "1970-01-01T00:00:00.000Z". One intentional quirk: when writing times to human-oriented files (e.g., .csv, .tsv, .json, .xhtml), ERDDAP uses the specified time_precision if it includes seconds and/or decimal seconds; otherwise, it uses seconds time_precision "1970-01-01T00:00:00Z" (for consistency and backwards compatibility). Thanks to Dominic Fuller-Rowell.
    • EDDGridFromNcFiles now supports reading String dataVariables.
    • .nc files written by griddap can now have String dataVariables.
    • GenerateDatasetsXml now includes more flush() calls to avoid the problem of information not being written to the files. Thanks to Thierry Valero.
    • The documentation for GenerateDatasetsXml was improved, notably to point out that the -i switch only works if you specify all the answers on the command line (e.g., script mode). And script mode is explained. Thanks to Thierry Valero.
    • ERDDAP no longer allows two variables in a dataset to have the same sourceName. (If someone did it before, it probably led to error messages.) As before, ERDDAP doesn't allow two variables in a dataset to have the same destinationName.

Changes in ERDDAP version 1.52 (released 2014-10-03)

  • New Features: (none)
  • Small changes/Bug Fixes:
    • Another (smaller) change to make ERDDAP faster.
    • Improvement to ISO 19115 files generated by ERDDAP: added newly recommended <gmd:protocol> values (information, search, OPeNDAP:OPeNDAP, ERDDAP:griddap, and ERDDAP:tabledap) within <gmd:CI_OnlineResource>. Thanks to Derrick Snowden and John Maurer.
    • Many small changes.
  • Things ERDDAP Administrators Need to Know and Do:
    • Bug fix: GenerateDatasetsXml.sh and DasDds.sh weren't in erddap.war for 1.48 and 1.50. Now they are. Thanks to Thierry Valero.
    • Small changes to some speed tests in TestAll to make them less susceptible to chance. Thanks to Terry Rankine.

Changes in ERDDAP version 1.50 (released 2014-09-06)

  • New Features: (none)
  • Small changes/Bug Fixes:
    • This ERDDAP should be much faster than recent versions.
  • Things ERDDAP Administrators Need to Know and Do: (nothing)

Changes in ERDDAP version 1.48 (released 2014-09-04)

  • New Features:
    • ERDDAP now always creates a tabular dataset, datasetID=allDatasets, which has a table of information about all of the datasets in this ERDDAP. It can be queried like any other tabular dataset. This is a useful alternative to the current system for getting information about datasets programmatically.
    • There are two new output file types for EDDTable and EDDGrid, .csv0 and .tsv0. They are comma- and tab-separated-value files that don't have lines with column names or units. The data starts on the first line. They are especially useful for scripts that just want one piece of information from ERDDAP.
  • Small changes/Bug Fixes:
    • Maps can now be made to longitudes in the range -720 to 720.
    • The new .ncml response File Type is available for all EDDGrid datasets. It returns the NCML (external link)-formatted description of the dataset (similar to a combined .dds + .das).
    • Bug fix: Saving tabular data to an .nc file was limited to 100,000 values per variable. Now it is just limited to 2 GB total file size. Thanks to Kevin O'Brien.
    • Bug fix: the saveAsMatlab methods now ensure that datasetIDs are converted to safe Matlab variable names. But I still strongly recommend that you create datasetIDs that are valid variable names: starting with a letter and then just using A-Z, a-z, 0-9, and _. See datasetID. Thanks to Luke Campbell.
    • Bug fix in EDDTableFromDatabase: With some types of databases, a NO_DATA response from the database led to a pointless 30 second delay in ERDDAP. Thanks to Greg Williams.
    • Bug fix: EDDGrid Make A Graph with Graph Type = lines (or markers or markers and lines) forced x axis variable to be time. Now it can be any axis. Thanks to Lynn DeWitt.
  • Things ERDDAP Administrators Need to Know and Do:
    • STRONGLY RECOMMENDED: Update Java
      This version of ERDDAP requires Java 7 or higher, but Java 7 will reach its end-of-life in April 2015 (soon!), so now is a good time to switch to Java 8. So Java 8 is STRONGLY RECOMMENDED. I test with Java 8. Note that Java 6 reached its end-of-life in February 2013 (no more security bug fixes!).
    • STRONGLY RECOMMENDED: Update Tomcat
      If you use Tomcat, please switch to the latest version of Tomcat. Tomcat 8 is designed to work with Java 8.
    • "ERDDAP" is no longer an acronym. Now it is just a name. I don't want the name to highlight ERD. I want ERDDAP to highlight your institution and your data.
    • PLEASE customize the appearance of your ERDDAP installation to highlight your institution and your data. With an hour's work, you can make nice improvements that will last forever.
    • In setup.xml, the <displayDiagnosticInfo> option is now always ignored and treated as if the value were false.
      RECOMMENDED: Remove the <displayDiagnosticInfo> tag and related info from your setup.xml.
    • In setup.xml, the default for <drawLandMask> was "over", but now it is "under", which is a better general default (works well with all datasets).
    • The GenerateDatasetsXml.sh and DadDds.sh Linux scripts now use bash instead of csh, and have the extension .sh. Thanks to Emilio Mayorga
    • GenerateDatasetsXml and DasDds now create their own log files (GenerateDatasetsXml.log and DasDds.log) and output files (GenerateDatasetsXml.out and DadDds.out) in bigParentDirectory/logs/, and never put their results on the clipboard.
    • GenerateDatasetsXml now supports a -i command line parameter which inserts the output into the specified file at a specified place. See the documentation. Thanks to Terry Rankine.
    • EDDTableFromDatabase now supports <columnNameQuotes></columnNameQuotes>, with valid values " (the default), ', or nothing. This character (if any) will be used before and after column names in SQL queries. Different types of databases, set up in different ways, will need different column name quotation marks.
    • Tabular latitude and longitude variables can now have customized long_name's, e.g., Profile Latitude. Previously, they could only be Latitude and Longitude.
    • From now on, specify "defaultDataQuery" and "defaultGraphQuery" as attributes in the dataset's global metadata (i.e., <addAtts>), not as separate <defaultDataQuery> and <defaultGraphQuery> tags. (Although, if you still specify them via the tags, ERDDAP will automatically create global attributes with the information.)

Changes in ERDDAP version 1.46 (released 2013-07-09)

  • New Features:
    • (None)
  • Small changes/Bug Fixes:
    • Bug fix: In EDDTableFromDatabase, in version 1.44 only, ERDDAP improperly quoted the database's table name in SQL statements. That is now fixed. Thanks to Kevin O'Brien.
  • Things ERDDAP Administrators Need to Know and Do:
    • If you don't modify the standard messages in messages.xml,
      delete [tomcat]/content/erddap/messages.xml .

      The default messages.xml file is now in the erddap.war file, not erddapContent.zip. So, you no longer need to manually update messages.xml .
    • If you do modify the messages in messages.xml, from now on, each time you update ERDDAP, either:
      • Make the same changes you made before to the new
        [tomcat]/webapps/erddap/WEB-INF/classes/gov/noaa/pfel/erddap/util/messages.xml.
        And this one time: delete [tomcat]/content/erddap/messages.xml .
      • Or, figure out what has changed in the new messages.xml (via diff), and modify your
        [tomcat]/content/erddap/messages.xml file accordingly.

Changes in ERDDAP version 1.44 (released 2013-05-30)

  • New Features:
    • Queries to EDDTable datasets now support &orderByMin(...) and &orderByMinMax(...) (which returns two rows in each group, with the minimum and maximum of the last orderBy value). Thanks to Lynn DeWitt.
    • There are two new tabledap file types: .ncCFHeader and .ncCFMAHeader (which return the ncdump-like header of the corresponding .ncCF and .ncCFMA file types). Thanks to Steve Hankin.
  • Small changes/Bug Fixes:
    • Bug fix: loading the .graph and .html web pages for datasets with lots of time values was slow because ERDDAP was slow when generating the time slider options. Now it is always fast. Thanks to Michael Barry, OOICI, and Kristian Sebastian Blalid.
    • Bug fix: In some EDDTable dataset types, the time constraints were not always handled correctly. Now they are. Thanks to John Maurer and Kevin O'Brien.
    • Bug fix: datasets wouldn't load when all of the subsetVariables were fixed value variables. Now they will. Thanks to Lynn DeWitt and John Peterson.
    • Change: now, all queries for just subset variables act as if &distinct() is part of the query.
    • Change: now, for queries that include &.jsonp=functionName, functionName MUST now be a series of 1 or more (period-separated) words. Each word must start with an ISO 8859 letter or "_" and be followed by 0 or more ISO 8859 letters, digits, or "_". Yes, this is more restrictive than JavaScript's requirements for function names.
    • The time axis on graphs now works well for longer time ranges (80 - 10000 years) and shorter time ranges (0.003 - 180 seconds).
    • ERDDAP is now more forgiving when parsing variations of ISO-8601-format time data.
    • There were many other small changes and bug fixes.
  • Things ERDDAP Administrators Need to Know and Do:
    • You MUST update to the latest version to be secure.
      ERDDAP underwent a security audit. There were some bugs and weaknesses. Version 1.44 includes several important security bug fixes and several changes to increase security and accessibility (e.g., for vision impaired users). Version 1.44 has passed the follow-up security audit. Thanks to all the good people at USGS and Acunetix who made this possible. (Shouldn't NOAA be doing this?)
    • The new EDDTableFromWFSFiles makes a local copy of all of the data from an ArcGIS MapServer WFS server and so the data can then be re-served quickly to ERDDAP users. Thanks to Christy Caudill.
    • The new EDDTableFromEDDGrid lets you create an EDDTable dataset from an EDDGrid dataset. Some common reasons for doing this are:
      • This allows the dataset to be queried with OPeNDAP selection constraints (which a user may have requested).
      • The dataset is inherently a tabular dataset.
      Thanks to OOICI, Jim Potemra, Roy Mendelssohn.
    • The variable name "depth" is now a special alternative to "altitude". The units must be some variant of "meters". The data values must be positive=down. ERDDAP is now fully aware of the meaning of "depth" and supports it wherever altitude is supported (e.g., as a component of a CF DSG cdm_data_type=profile dataset). A dataset must not have both "depth" and "altitude" variables.
    • In your datasets.xml, please remove any uses of <att name="cdm_altitude_proxy">depth</att> since depth is now a special alternative to altitude and so doesn't need to be specially identified.
    • In your datasets.xml, please remove any uses of <altitudeMetersPerSourceUnit>, except for EDDTableFromSOS.
      When the value is 1, just delete it.
      When the value is -1, consider changing the variable name to depth.
      For other values, add to <addAttributes>, for example,:
      <att name="scale_factor" type="float">-1</att>
    • All datasets now support
      • <defaultDataQuery> which is used if .html is requested with no query.
        • You will probably rarely need to use this.
        • For griddap datasets, a common use of this is to specify a different default depth or altitude dimension value (e.g., [0] instead of [last]).
          In any case, you should always list all of the variables, always use the same dimension values for all variables, and almost always use [0], [last], or [0:last] for the dimension values.
          For example:
          <defaultDataQuery>u[last][0][0:last][0:last],v[last][0][0:last][0:last]</defaultDataQuery>
        • For tabledap datasets, the most common use of this is to specify a different default time range (relative to now, e.g., &time>=now-1day).
          Remember that requesting no data variables is the same as specifying all data variables, so usually you can just specify the new time constraint.
          For example:
          <defaultDataQuery>&amp;time&gt;=now-1day</defaultDataQuery>
      • <defaultGraphQuery> which is used if .graph is requested with no query.
        • You will probably rarely need to use this.
        • For gridddap datasets, the most common use of this is to specify a different default depth or altitude dimension value (e.g., [0] instead of [last]) and/or to specify that a specific variable be graphed.
          In any case, you will almost always use [0], [last], or [0:last] for the dimension values.
          For example:
          <defaultGraphQuery>temp[last][0][0:last][0:last]&amp;.draw=surface&amp;.vars=longitude|latitude|temp</defaultGraphQuery>
        • For tabledap datasets, the most common uses of this are to specify different variables to be graphed, a different default time range (relative to now, e.g., &time>=now-1day) and/or different default graphics settings (e.g., marker type).
          For example:
          <defaultGraphQuery>longitude,latitude,seaTemperature&amp;time>=now-1day&amp;.marker=1|5</defaultGraphQuery>
      Remember that you need to XML-encode or percent-encode (either one, but not both) the default queries since they are in an XML document. For example, & becomes &amp; , < becomes &lt; , and > becomes &gt; .
      And please check your work. It is easy to make a mistake and not get what you want.
      Thanks to Charles Carleton, Kevin O'Brien, Luke Campbell, and others.
    • EDDGridFromDap, EDDGridFromErddap, and EDDTableFromEDDGrid have a new system to deal with datasets that change frequently (as often as roughly every 0.5 s). Unlike ERDDAP's regular, proactive system for completely reloading each dataset, this optional additional system is reactive (triggered by a user request) and incremental (just updating the information that needs to be updated). For example, if a request to an EDDGridFromDap dataset occurs more than the specified number of milliseconds since the last update, ERDDAP will see if there are any new values for the leftmost (usually "time") dimension and, if so, just download those new values before handling the user's request. This system is very good at keeping a rapidly changing dataset up-to-date with minimal demands on the data source, but at the cost of slightly slowing down the processing of some user requests. See <updateEveryNMillis>
      Thanks to Michael Barry and OOICI.
    • EDDGridFromNcFiles, EDDTableFromNcFiles, and EDDTableFromNcCFFiles now support NcML .ncml source files in place of .nc files. Thanks to Jose B Rodriguez Rueda.
    • For EDDGridAggregateExistingDimension, ERDDAP supports a new serverType="dodsindex" option for the serverType attribute of the <sourceUrls> tag. This works with web pages that have lists of files within <pre></pre> and often beneath the page title "DODS Index of /...". An example is http://www.marine.csiro.au/dods/nph-dods/dods-data/bl/BRAN2.1/bodas/ (external link).
    • For EDDTableFromSOS now supports an optional tag
      <sosServerType>serverType</sosServerType>
      so you can specify the type of SOS server (so ERDDAP doesn't have to figure it out). Valid values of <serverType> are IOOS_NDBC, IOOS_NOS, OOSTethys, and WHOI (a newly supported serverType). See EDDTableFromSOS. Thanks to Derrick Snowden and Janet Fredericks.
    • All EDDGridFrom...Files, EDDTableFrom...Files, EDDGridCopy, and EDDTableCopy now support an optional tag
      <fileTableInMemory>true</fileTableInMemory> (The default is false.)
      which can tell ERDDAP to keep the fileTable (with information about each source data file) in memory instead of just on the disk (the default). Keeping the fileTable in memory speeds up requests for data (especially if there are >1000 source data files), but uses more memory. If you set this to true for any dataset, keep an eye on the Memory: currently using line at yourDomain/erddap/status.html to ensure that ERDDAP still has plenty of free memory. Thanks to Fredrik Stray.
    • EDDTableFromASCIIFiles now supports <charset>. The two most common charsets (case sensitive!) are ISO-8859-1 (the default) and UTF-8.
    • Recommended: in setup.xml, within <startHeadHtml>, please change <html> into
      <html lang="en-US"> (or a different language code (external link) if you have translated messages.xml).
    • setup.xml has new optional tags to disable parts of ERDDAP:
      • <convertersActive>false</convertersActive> <!-- the default is true -->
      • <slideSorterActive>false</slideSorterActive> <!-- the default is true -->
      • <wmsActive>false</wmsActive> <!-- the default is true -->
      In general, we recommend against setting any of these to false.
    • GenerateDatasetsXml now writes results to bigParentDirectory/logs/generateDatasetsXmlLog.txt, not log.txt. Thanks to Kristian Sebastian Blalid.
    • GenerateDatasetsXml now makes a good suggestion for the <reloadEveryNMinutes>. Thanks to the NOAA UAF project.
    • Many small improvements to GenerateDatasetsXml. Thanks to the NOAA UAF project.

Changes in ERDDAP version 1.42 (released 2012-11-26)

  • New Features:
    • (No major new features.)
  • Things ERDDAP Administrators Need to Know and Do:
    • If you are upgrading from ERDDAP 1.38 or 1.40, there were no changes that require you to make changes to your configuration files (but you must use the new messages.xml file).
    • ERDDAP once again can run with Java 1.6. (ERDDAP v1.40 required Java 1.7.) We still strongly recommend using the latest version of Java 1.7.
    • A new dataset type, EDDTableFromAwsXmlFiles, can read data from a set of Automatic Weather Station (AWS) XML data files. Thanks to Lynn Dewitt and the Exploratorium.
  • Small changes/Bug Fixes:
    • Adjusted to changes to the NDBC SOS source data servers.
    • Adjusted to changes to the NOS COOPS ASCII services.
    • Made several small changes and bug fixes.

Changes in ERDDAP version 1.40 (released 2012-10-25)

  • New Features:
    • There is a new output file format for tabledap datasets: .ncCFMA, which saves the requested data in a .nc file which conforms to the CF Discrete Sampling Geometries (external link) Multidimensional Array options, and which therefore conforms to the NODC templates (external link) for storing this type of data. Thanks to NODC.
    • tabledap requests can now include time constraints such as &time>now-5days. See the documentation. Thanks to James Gosling.
  • Things ERDDAP Administrators Need to Know and Do:
    • If you are upgrading from ERDDAP 1.38, there were no changes that require you to make changes to your configuration files (but you must use the new messages.xml file).
    • ERDDAP public releases and internal milestones are available via ERDDAP on GitHub (external link). For more information, see the Wiki (external link) for the ERDDAP project as well as the more general ERDDAP Programmer's Guide. (This was announced separately a few weeks after the ERDDAP 1.38 release.)
    • GenerateDatasetsXml has been improved.
      • The script was revised so it should work correctly all all Linux computers (not just a few).
      • It now adds creator_name, creator_email, and creator_url whenever possible.
      • Many other small improvements.
    • Refined how ERDDAP deals with time.
      • Internally, ERDDAP now handles times at millisecond precision (not seconds).
      • You can now optionally specify the time precision for a given dataset, see time_precision. For example, you might set a dataset to display time values with date precision (e.g., 1970-01-01).
      • Your current datasets will use the default settings, so they are unaffected by these changes and will continue to display time with seconds precision.
      Thanks to Servet Cizmeli and Philip Goldstein.
    • EDDTableFromNcCFFiles is a new dataset type which you can use in your datasets.xml file. It can read data from any of the numerous file formats defined by the CF Discrete Sampling Geometries (external link) conventions. Thanks to NODC and special thanks to Kyle Wilcox for making sample files for the huge number of valid DSG file formats and for making them publicly available.
  • Small changes/Bug Fixes:
    • Expanded the quickRestart system to all relevant EDDGrid and EDDTable subclasses.
    • Improved documentation, especially related to how to use griddap and tabledap from various client software.
    • Changed advanced search to support minTime and/or maxTime expressed as epochSeconds. Thanks to Lynn Dewitt.
    • Changed .htmlTable output to display urls and email addresses as links.
    • Added "rel=" and "rev=" to relevant <a href> tags. Thanks to Pat Cappelaere from the OGC REST project.
    • Improved protection against unrealistically large data requests, notably within tabledap, where it is a harder problem.
    • Moved more messages to messages.xml.
    • Made speed improvements.
    • Fixed EDDGridFromFiles to allow descending sorted axes. Thanks to Maricel Etchegaray.
    • Removed references to iGoogle since it will be discontinued.
    • Made several small changes and bug fixes.

Changes in ERDDAP version 1.38 (released 2012-04-21)

  • New Features:
    • ISO 19115 and FGDC - ERDDAP can automatically generate ISO 19115 and FGDC XML metadata files for each dataset. Links to the files are visible on every list of datasets (e.g., from Full Text Search) and also in Web Accessible Folders (WAF) (see the FGDC WAF and ISO 19115 WAF). Thanks to Ted Habermann, Dave Neufeld, and many others.
    • Full Text Searches for Datasets now support -excludedWord and -"excluded phrase" . Thanks to Rich Signell.
    • Searches for datasets now return results a page at a time. The default uses the parameter string: page=1&itemsPerPage=1000, but you can change the values in the URL of your request. Thanks to Steve Hankin and the UAF project.
    • OpenSearch - ERDDAP now supports the OpenSearch 1.1 standard for searching for datasets. Among other things, this allows catalog aggregation websites to do distributed searches (passing a search request to each catalog that it knows about).
    • Comma Separated Value (CSV) Files - ERDDAP now generates CSV files with just a comma between values (which Excel prefers), instead of comma+space. Thanks to Jeff deLaBeaujardiere.
    • Million Datasets - Several changes were made to support ERDDAPs having a huge number of datasets, perhaps even a million. Thanks to Steve Hankin and the UAF project.
  • Things ERDDAP Administrators Need to Know and Do:
    • A quick restart system allows ERDDAP to restart much faster.
      Please add this to your setup.xml file right after </datasetsRegex>:
            <!-- If true, when you start up ERDDAP, some types of datasets (e.g., 
            EDDGridFromDap) will used cached information (.dds, .das, etc.) to reload
            very quickly, without contacting the remote server.  The dataset's age 
            will be based on when the dataset was reloaded last.  Normally this 
            should be true (the default), but set it to false if you want to bypass 
            the cached information.
            <quickRestart>true</quickRestart>
    • Full text searches for datasets can now be done with the Lucene search engine (although we recommend the original search engine if you have fewer than 10,000 datasets) or the original search system.
      Please add this to your setup.xml file right after </displayDiagnosticInfo>:
            <!-- ERDDAP lets you choose between two search engines for full text searches:
            * original (the default) - is the best choice if your ERDDAP has fewer 
              than about 10,000 datasets.  It is very robust and trouble free. 
            * lucene - is the best choice for more than about 10,000 datasets.
              The advantages are that with any number of datasets it works fast 
              and uses very little memory.
              But there are many things that might go wrong with individual 
              queries and with the whole system. 
              And although its behaviour (the datasets it finds and the order that
              it ranks them) is almost identical to the original search engine,
              it has a few quirky, subtle, small differences.
            -->
            <searchEngine>original</searchEngine>
    • In setup.xml, you can/should now add two new categories to the comma-separated list of <categoryAttributes>:
      • global:keywords (add it right after global:institution) -- a new special case that parses a comma-separated list of keywords from the global keywords attribute to make a separate entry for each keyword.
      • variableName (add it at the end) -- a new special case that categorizes each of the dataVariable destinationNames.
    • In setup.xml, you can (but why?) tell ERDDAP not to offer FGDC and/or ISO 19115 metadata for any dataset by including
      <fgdcActive>false</fgdcActive>
      <iso19115Active>false</iso19115Active>

      The default values for these settings is true.
    • In datasets.xml, please consider improving the metadata for your datasets. ERDDAP now automatically generates ISO 19115 and FGDC XML metadata files for each dataset based on the dataset's metadata.
      So, good dataset metadata leads to good ERDDAP-generated ISO 19115 and FGDC metadata.
      See the new documentation for the many new RECOMMENDED Global Attributes.
    • In datasets.xml, if you want to tell ERDDAP to use a pre-made FGDC and/or ISO 19115 file that is somewhere on the server's file system instead of having ERDDAP generate these files, use:
      <fgdcFile>fullFileName</fgdcFile>
      <iso19115File>fullFileName</iso19115File>

      If fullFileName="" or the file isn't found, the dataset will have no FGDC and/or ISO 19115 metadata. So this is also useful if you want to suppress the FGDC and/or ISO 19115 metadata for a specific dataset.
    • In datasets.xml, for all EDDGridSideBySide and EDDGridAggregateExistingDimension datasets, make certain that child datasets have different datasetIDs than their parent datasets and than the other children. (For example, you could follow George Foreman's simple but effective system for naming his children.) If any names in a family are exactly the same, the dataset will fail to load (with the error message that the values of the aggregated axis are not in sorted order).
    • In datasets.xml, there were some changes to the list of valid ioos_category metadata values:
      • "pCO2" was changed to "CO2".
      • "Physical Oceanography" was added.
      • "Soils" was added.
    • In datasets.xml, ERDDAP no longer allows '.' in a datasetID. It was allowed but discouraged. (Sorry)
    • In datasets.xml, the setup for EDDTableFromThreddsFiles and EDDTableFromHyraxFiles has changed slightly because both classes were just rewritten to be more efficient (both classes now always make a local copy of all of the remote data files). See the documentation for setting up these classes: EDDTableFromHyraxFiles and EDDTableFromThreddsFiles. In particular, see the revised comments about <fileDir> (now irrelevant) and <sourceUrl> (now essential). Also, you should never wrap this class in EDDTableCopy for efficiency.
    • In datasets.xml, if you use EDDTableFromDatabase with an Oracle database, you should include a connectionProperty such as
      <connectionProperty name="defaultRowPrefetch">4096</connectionProperty>
      to specify how many rows of data to fetch at one time because the default is 10, which is horribly inefficient. See the Oracle documentation (external link). MySql and PostgreSQL seem to have better defaults for this setting. Thanks to Kevin O'Brien.
    • If you use EDDTableFromDatabase, see the improved "Speed" documentation for additional suggestions to improve performance. Thanks to Kevin O'Brien.
    • In datasets.xml, for all EDDTable... datasets, in the Conventions and Metadata_Conventions global attributes, please refer to CF-1.6 (not CF-1.0, 1.1, 1.2, 1.3, 1.4, or 1.5), since CF-1.6 is the first version to include the changes related to the Discrete Sampling Geometry.
    • Programmers that are compiling the ERDDAP code need to add lib/lucene-core.jar to the list of jar files in their javac and java command line paths.
    • ERDDAP has a new service to convert a CF Standard Name to/from a GCMD Science Keyword. You may find this useful when generating global keywords metadata for the datasets in your ERDDAP.
    • Dealing with Bots - Please read this advice to prevent bots from crawling your ERDDAP in a stupid way.
    • Translation - The text on ERDDAP's web pages is now mostly in messages.xml and so suitable for translation to different languages (e.g., German, French). The messages now often use MessageFormat for formatting, also to aid in making translations. If you are interested in doing a translation, please email bob dot simons at noaa dot gov .
    • Sample datasets.xml - There were several small but significant errors in the sample datasets.xml. If you use those datasets, please get the newer versions from the new sample datasets.xml in the new erddapContent.zip. Thanks to James Wilkinson.
    • Git - I will try hard to make ERDDAP a GitHub project ASAP after this release.
  • Small changes/Bug Fixes:
    • A new palette, OceanDepth, is useful for depth values (positive is down), e.g., 0 (shallow) to 8000 (deep).
    • The .kml output from tabledap uses a better marker icon (it isn't fuzzy). And hovering over a marker now makes it bigger.
    • EDDTableFromFiles - In the last upgrade, the new netcdf-java library had tighter restrictions for variable names in .nc files. That caused problems for EDDTableFromFiles if a variable's sourceName had certain punctuation characters. EDDTableFromFiles is now modified to avoid that problem. Thanks to Thomas Holcomb.
    • The .subset page now supports 0/10/100/1000/10000/100000 instead of a check box for Related Data. The tooltip warns that 100000 may cause your browser to crash. Thanks to Annette DesRochers, Richard (Abe) Coughlin, and the IOOS Biological Project.
    • .../erddap/info/datasetID/index.html web pages now show urls and email addresses as clickable links. Thanks to Richard (Abe) Coughlin and the IOOS Biological Project.
    • Bug fix: In tabledap, for datasets with altitudeMetersPerSourceUnit < 0, queries with altitude constraints were handled incorrectly. Thanks to Kyle Wilcox.
    • Bug fix: EDDGridAggregateFromExistingDimension now supports more diverse TDS URLs. Thanks to ?

Changes in ERDDAP version 1.36 (released 2011-08-01)

Changes in ERDDAP version 1.34 (released 2011-06-15)

  • Changes:
    • Bug fix: Fixed memory leak that occurred on some 64-bit Java installations.
    • Bug fix: ERDDAP now correctly sets these global attributes when the latitude dimension's values ranged from high to low: geospatial_lat_min, geospatial_lat_max, Southernmost_Northing, Northernmost_Northing.

      Note that actual_range is unchanged: it may have low,high values or high,low values, since it is intended to indicate the range and the order of storage.

    • Small changes.
    • ERDDAP administrators don't need to make any changes to their setup.xml or datasets.xml.

Changes in ERDDAP version 1.32 (released 2011-05-20)

  • Changes:
    • Support for the newly ratified, CF Discrete Sampling Geometries (which unfortunately is not yet available online), which replaces the proposed CF Point Observation Conventions.
      ERDDAP users will see that cdm_feature_type=Station is replaced by TimeSeries and there are small changes to the files created for the .ncCF file type (flat_dimension is now called sample_dimension).
      ERDDAP administrators will need to make these changes in datasets.xml:
      • cdm_data_type=Station should be changed to cdm_data_type=TimeSeries.
      • cdm_data_type=StationProfile should be changed to cdm_data_type=TimeSeriesProfile.
      • cdm_station_variables should be changed to cdm_timeseries_variables.
      • cf_role=station_id should be changed to cf_role=timeseries_id.
    • New ioos_category options: "Colored Dissolved Organic Matter", "pCO2", "Stream Flow", "Total Suspended Matter".
    • Possible solution to a possible memory leak on 64-bit Java. [It didn't work.]
    • Small changes.

Changes in ERDDAP version 1.30 (released 2011-04-29)

  • New Features:
    • Support for 64-bit Java. When used with a 64-bit Java, ERDDAP can now use much more heap memory and handle many more simultaneous requests.
    • Support for .nc file requests up to 2GB (even without 64-bit Java) via better use of ERDDAP's handling of data in chunks.
    • Many 2X speed improvements in the code and 2X speed ups from Java 1.6 make ERDDAP 2X to 4X faster than before.
    • Memory saving improvements significantly lower ERDDAP's base memory usage.
    • For tabular datasets, ERDDAP is now fully aware of a dataset's cdm_data_type, and how the data maps to the CDM type. See the CF Discrete Sampling Geometries specification (external link). Perhaps someday soon, that Word file will be converted to .html and replace the current "OBSOLETE" information on that web page. Thanks to the NOAA UAF project.
    • For most EDDTable datasets, a new output file type option, .ncCF, creates Contiguous Ragged Array .nc files which conform to the latest version of the CF Discrete Sampling Geometries conventions (external link). These files are structured to reflect the CDM data type of the dataset. Since the proposed conventions just changed, as of this writing, the netcdf-java library does not yet support reading the file formats created by ERDDAP and interpreting them as CDM data files. It probably will soon. Thanks to the NOAA UAF project.
    • The View : Distinct Data option on the .subset web page is now a drop-down list that lets users specify the maximum number of rows of distinct data to be viewed (default = 1000). This change, and others, allow ERDDAP to work with datasets that have very large numbers of rows of distinct data. (The number of unique values for any single variable is still an issue, but it can be pretty high (20,000?) before the .subset and other web pages load really slowly.) Thanks to the NOAA UAF project.
    • .subset web pages have a new option: View Distinct Data Counts. Thanks to the GTOPP project.
    • To aid users, the distinct values (e.g., station names) are now shown on the Make-A-Graph and Data Access Forms. Thanks to the NOAA UAF project.
    • .transparentPng requests now support all types of graphs and data representations. It draws just the data -- no axes, legends, landmask, or anything else. This makes it possible to make images as layers of transparentPngs. If &.size=width|height is specified in the query (recommended), it is honored. The default is 360x360 pixels. The only exception is EDDGrid &.draw=surface, where the default (as before) is an image with ~1/pixel per datam (up to 3000 x and y pixels). Thanks to Fred Hochstaedter.
    • The WMS web pages now show the color bar for the dataset's variable(s). Thanks to Emilio Mayorga and others.
  • Things ERDDAP Administrators Need to Know and Do:
    • This release involves a lot of changes. They are all important. Please be patient and work through all of the changes listed below.
    • This version is being pushed out earlier than intended to deal with some Java security bugs. Unfortunately, several features/fixes intended for this ERDDAP version are not in this version. Sorry. Hopefully, the next version will be relatively soon (and much easier to upgrade to).
    • To avoid several security bugs in Java 6 update 23 and below, download (external link) and install the latest version of Java (Java 6 update 24 or higher). If you have a 64-bit operating system, please get a 64-bit version of Java.
    • If you are using Tomcat 5, you MUST upgrade to Tomcat 6 or 7 (preferred). If you are using Tomcat 6, consider upgrading to Tomcat version 7.
    • Please follow all of the instructions for setting up a new ERDDAP, but where relevant, you will be copying files from your old installation to the new installation, notably the [tomcat]/content/erddap directory and files. As part of that, note the new Tomcat setup recommendations.
    • The default erddap.css is now included in the erddap.war file.
      • To use the default erddap.css, delete your old [tomcat]/content/erddap/images/erddap.css .
      • If you modified [tomcat]/content/erddap/images/erddap.css, and want to keep using it: just leave it in place and replace the <input> section with:
        /* Small input items let more be shown on one screen
        (esp. Chrome and Safari). Google Chrome and Safari have
        default margin 2px, while others are 0. This sets all to 0.
        .skinny is used e.g., for the buttons above the image on
        a Make A Graph page. */
        input[type=button], input[type=submit], button {
          margin:0px; padding:0px 3px; }
        input[type=checkbox], input[type=password],
          input[type=text], select, textarea {
          margin:0px; padding:0px; }
        input[type=radio] {margin:0px 2px; padding:0px; }
        input.skinny {padding:0px 1px; }
    • In your [tomcat]/content/erddap/setup.xml:
      • Replace the comments and tags related to <partialRequestMaxBytes> and <partialRequestMaxCells> with
        <!-- When possible (and it isn't always possible),
        ERDDAP breaks source data requests into chunks to
        conserve memory. See the description of these tags in
        messages.xml. You can override the default chunk sizes
        here with
        For grids:
         <partialRequestMaxBytes>100000000</partialRequestMaxBytes>
        For tables:
         <partialRequestMaxCells>100000</partialRequestMaxCells>
        -->
      • Replace the comments related to <categoryAttributes> and consider modifying the tag's value:
        <!-- This is the comma-separated list (recommended:
        in alphabetical order) of the global attribute and
        variable attribute names which will be used to
        categorize the datasets and shown to clients at urls
        like .../erddap/categorize/ioos_category/index.html
        (ioos_category is unusual, but is used at ERD).
        If an attribute is a global attribute, identify it by
        prefixing it with "global:".
        -->
        <categoryAttributes>global:institution, ioos_category,
        long_name, standard_name</categoryAttributes>

         
        Individual <categoryAttributes> that are global attributes now MUST be identified via the prefix global: (e.g., global:institution). Others attributes are assumed to be variable attributes (e.g., standard_name). Also, institution values (the only ones) were left in original case. Now all category values are converted to lower case.
    • In your [tomcat]/content/erddap/datasets.xml:
      • Big change: ERDDAP has new requirements related to a tabular dataset's cdm_data_type. Notably, each dataset MUST have the correct metadata and variables related to the cdm_data_type. If not, the dataset won't load and will throw an error. See the documentation for cdm_data_type.
      • FYI: There is a new dataset type: EDDTableFromAsciiServiceNOS.
      • FYI: There are three newly allowed ioos_category options: Hydrology, Quality (e.g., for quality flags), and Statistics (e.g., mean).
      • For EDDTableFrom...Files datasets, remove any <nDimensions> tags. They are no longer needed or used.
      • For variables with destinationName=altitude, ERDDAP no longer forces the long_name to be Altitude. Please go through your datasets.xml and repeatedly search for <destinationName>altitude and add to that variable's <addAttributes>:
          <att name="long_name">Altitude</att>
        (or a slightly different long_name in special cases).
      • Optional: All EDDTableFromFiles subclasses support variable sourceName=global:... to convert global metadata from each file into a data variable. Thanks to Lynn DeWitt.
    • EDDTableFromDatabase users - ERDDAP comes with a new JDBC 4 driver for Postgres. For other databases, check the web for the lastest JDBC .jar file for your database. Since ERDDAP now uses Java 1.6+, JDBC 4 (not 3) is probably recommended.
    • FYI
      • EDDGridFrom...Files and EDDTableFrom...Files datasets now store the fileTable information in
        [bigParentDirectory]/datasetInfo/[datasetID]/*.nc files.
        Also, EDDTable datasets now store the subset information in
        [bigParentDirectory]/datasetInfo/[datasetID]/*.nc files. These files used to be
        [bigParentDirectory]/datasetInfo/[datasetID].*.json files.
        The old files will be deleted automatically when ERDDAP starts up. Or, you can delete all files (but leave the empty subdirectories) in [bigParentDirectory]/datasetInfo/.
      • I worked on a new EDDTableFromNcCFFiles which would read data from local and remote files using the proposed, new CF Point Observation Conventions. But it isn't in this release. There are problems in the netcdf-java libraries related to some methods for reading these files. And there were some very recent changes to the proposed CF Point Observation Conventions. When the netcdf-java library is fixed and updated to the latest proposal, I will resume work on this.
      • Running ERDDAP on Windows may have problems: notably, you may see in the [bigParentDirectory/logs/log.txt file that ERDDAP is sometimes unable to delete and/or rename files quickly. This is due to antivirus software (e.g., from MacAfee and Norton) which is checking the files for viruses. If you run into this problem (which can be seen by error messages in the log.txt file like "Unable to delete ..."), changing the antivirus software's settings may partially alleviate the problem.
        If the ERDDAP in Windows is just a test running on your desktop, this is just an annoyance.
        If the ERDDAP in Windows is your public ERDDAP, consider switching to a Linux server.
    • Slow First Startup - The first time you run ERDDAP after upgrading, ERDDAP may be slow to load the datasets. The way ERDDAP stores information about aggregated files has changed, so ERDDAP will need to re-read some info from all of those files. That will take time.
    • Errors on Startup - Given the changes related to cdm_data_type, it is likely that some of your datasets won't load and will throw errors. Carefully read the Daily Report email that ERDDAP sends you when ERDDAP is finished starting up. It will have a list of datasets that didn't load (at the top) and the reason they didn't load (near the bottom).
    • If you get stuck or have other questions, email the details to me: bob.simons at noaa.gov.
    • Programmers - If you write Java programs that run ERDDAP code, you need to change some of the command line parameter references:
      • Change joda-time-1.6.2.jar to joda-time.jar
      • Change the Postgres JDBC .jar reference to postgresql.jdbc.jar
  • Small Changes and Bug Fixes:
    • Improved connection handling to avoid hung threads.
    • Improved concurrency practices to handle nearly simultaneous identical requests more efficiently.
    • ERDDAP now uses netcdfAll-4.2.jar (renamed to netcdfAll-latest.jar). This switch necessitated several internal changes and causes a few small external changes, e.g., changes to how grib files are read and tiny changes to the .ncHeader output.
    • New feature: [erddap]/convert/fipscounty.html converts FIPS county codes to/from county names.
    • On maps, state boundaries are now dark violet, so they stand out a better on all background colors.
    • Tabular .kml output again uses a circular icon to mark points (not the airplane icon Google recently switched to).
    • The erdCalcofi datasets were re-arranged and are now served from local files (faster).
    • GenerateDatasetsXml fromThreddsCatalog now creates a results file:
      [tomcat]/webapps/erddap/WEB-INF/temp/EDDGridFromThreddsCatalog.xml . Thanks to Kevin O'Brien.
    • GenerateDatasetsXml fromThreddsCatalog now tries to remove unnecessary port numbers from the source URLs (e.g., :8080 and :8081 can sometimes be removed). Thanks to NOAA central's security team.
    • For .subset web pages, the Map of Distinct Data now has a variable lat lon range.
    • Several lists in ERDDAP (e.g., the table which shows all of the datasets) were sorted so that A..Z sorted before a..z. Now they sort in a case-insensitive way.
    • Small changes to the .subset web pages, including: units are now indicated.
    • GenerateDatasetsXml and DasDds no longer throw an exception if unable to put the results on the system clipboard or displayInBrowser. Thanks to Eric Bridger and Greg Williams.
    • Bug fix: When datasets are loaded, ERDDAP now removes or adjusts the geospatial global attributes. Thanks to Charles Carleton.
    • Bug fix: String2.getClassPath() now properly percent-decodes the classPath (notably, on Windows, spaces in the file name appeared as %20). This affected ERDDAP EDStatic calling SSR.getContextDirectory() and finding content/erddap. Thanks to Abe Coughlin.
    • Bug fix: in EDDTableFromFiles related to getDataForDapQuery handling of distinct() requests. Thanks to Eric Bridger.
    • Bug fix: tabledap requests didn't properly handle altitude constraints when the dataset's altitudeMetersPerSourceUnit was -1. Thanks to Eric Bridger.
    • Bug fix: EDDTableFrom...Files datasets now correctly handle requests which include =NaN and !=NaN.

     

Changes in ERDDAP version 1.28 (released 2010-08-27)

  • New Features: none.
  • Things ERDDAP Administrators Need to Know and Do: none.
  • Bug Fix: Fix a programming mistake (only in ver 1.26) that made ERDDAP very slow.
     

Changes in ERDDAP version 1.26 (released 2010-08-25)

  • New Features: none.
  • Things ERDDAP Administrators Need to Know and Do:
    • From your [tomcat]/content/erddap/setup.xml,
      • In <legal>, on a new line below [standardDataLicenses], insert [standardContact]. [standardContact] refers to the <adminEmail> specified higher up in setup.xml.
      • Remove <tableCommonBGColor> and <tableHighlightBGColor>.
      • Recommended: Change <endBodyHtml> to
        <endBodyHtml><![CDATA[
        <br>&nbsp;
        <hr>
        ERDDAP, Version &erddapVersion;
        <br><a href="&erddapUrl;/legal.html">Disclaimers</a> |
        <a href="&erddapUrl;/legal.html#privacyPolicy">Privacy Policy</a> |
        <a href="&erddapUrl;/legal.html#contact">Contact</a>
        </body>
        ]]></endBodyHtml>
    • Required: To your [tomcat]/content/erddap/images/erddap.css and erddapAlt.css, add at the bottom:
      /* This is used on the /info/[datasetID]/index.html pages to highlight a row or cell. */
      tr.highlightBGColor {background-color:#cceecc; }
      td.highlightBGColor {background-color:#cceecc; }
  • Bug Fixes and Small Changes:
    • Bug fix: in some situations, forms didn't work in some versions of Internet Explorer. Thanks very much to Greg Williams.
    • Bug fix: The Make A Graph buttons didn't work if the dataset was from a remote ERDDAP.
    • Bug fix: WMS sometimes didn't work if the dataset was from a remote ERDDAP.
    • Many small changes and bug fixes.

     

Changes in ERDDAP version 1.24 (released 2010-08-06)

  • New Features:
    • New Subset web pages use faceted search to select subsets of tabular datasets. Thanks to POST.
    • New Advanced Search combines all of the other search options and adds longitude, latitude, and time bounding boxes. Thanks to Ellyn Montgomery. (Sorry for the delay.)
    • New Convert Time web page and service let you convert numeric times to/from ISO string times.
    • New Convert Units web page and service let you convert UDUNITS to/from UCUM units. Thanks to NOAA IOOS SOS.
    • If a tabledap request includes &units("UCUM"), the units names will be converted from original names (usually UDUNITS) to UCUM (external link) units names. This only affects units *names*, not data values. Thanks to NOAA IOOS SOS.
    • Improvements to Make A Graph web pages and graphs and maps:
      • If the graph is a map, there are new Make A Graph buttons to zoom in/out and a new option to click to change the map's center point. Thanks to POST.
      • Filter settings added near the bottom. Thanks to Greg Williams.
      • The built in coastline data files were updated to GSHHS v2.0. Thanks to POST.
      • Maps now include lakes and rivers. Thanks to POST. (Sorry, the Sacramento River Delta is missing because neither the coastline data nor the lake/river dataset deals with it.)
      • The built in pscoast-derived nation/state files were updated. Thanks to POST.
      • Topography.cpt was modified slightly. (Sorry if this adversely affects you.) Thanks to POST.
      • In griddap's Make A Graph, if a user changes a variable, the form is automatically resubmitted so that the axisVariables' showStartAndStop always reflects the graph variables. Thanks to Joaquin Trinanes.
      • For png and pdf image URLs:
        • New &.land=value, where value can be "under" (show topography) or "over" (just show bathymetry). If not specified, the default is set by drawLandMask in datasets.xml or setup.xml. Thanks to POST.
        • New: lines in the legend that are too long are automatically broken into multiple lines. Thanks to POST.
      • For png image URLs:
        • New &.legend=value, where value can be "Bottom" (default), "Off" or "Only". This lets you include the legend, exclude the legend, or get only the legend. Thanks to Cara Wilson.
        • New &.trim=nPixels leaves a border of nPixels (e.g., 10) at the bottom of the image. It is applied after .legend=Off. Thanks to Cara Wilson.
        • New &.size=width|height lets you specify the width and height for the image, in pixels.
    • New output file formats:
      • .csvp and .tsvp - like .csv and .tsv, but with "(units)" appended to column names on first line.
      • .odvTxt - makes a .txt file that simplifies getting data into Ocean Data View (ODV) (external link).
      • .esriCsv - makes a .csv file suitable for import in ESRI's ArcGIS. (tabular datasets only)
      Thanks to Jan Mason, Jeff de La Beaujardiere, and NOAA IOOS SOS project.
    • GUI improvements to the Categorize web pages. Also, the categorize values (other than institution) are now all lower case. Non-lower case requests are accepted (redirected) for backwards compatibility. Thanks to Roy Mendelssohn.
    • Error messages are now even shorter and more oriented to users. Thanks to Greg Williams.
    • An internal change which greatly reduces ERDDAP's base memory usage.
    • Many new features which are only relevant to the POST project.
  • Things ERDDAP Administrators Need to Know and Do: There are lots of changes. Sorry. But each one brings some nice benefit.
    • Big changes to GenerateDatasetXml - it now often asks more questions (see the relevant datasetTypes information) and now always generates essentially ready-to-use content for datasets.xml. You are still responsible for the setup, so you should still review the datasets.xml content before using it. A human putting effort into the project will always do better than a computer program. Thanks to the UAF project.
    • REQUIRED: In setup.xml, you must revise the WMS section. It should now include these tags (but feel free to change the values):
      <!-- These default accessConstraints, fees, and keywords are used 
      by the SOS, WCS, and WMS services.
      They can be overridden by "accessConstraints", "fees", "keywords" 
      attributes in a dataset's global metadata.
      If a dataset that has an "accessibleTo" tag doesn't override 
      "accessConstraints", then the default for "accessConstraints" is the
      "accessRequiresAuthorization" value.  
      -->
      <accessConstraints>NONE</accessConstraints>
      <accessRequiresAuthorization>only accessible to authorized
      users</accessRequiresAuthorization>
      <fees>NONE</fees>
      <keywords>Earth science, oceans</keywords> 
      
      <!-- This appears on the erddap/legal.html web page after the 
      General Disclaimer. 
      You can replace any of the [standardParts] with your own HTML. -->
      <legal><![CDATA[
      [standardDisclaimerOfEndorsement]
      [standardDisclaimerOfExternalLinks]
      [standardPrivacyPolicy]
      [standardDataLicenses]
      ]]></legal>
      
      <!-- Specify the default units standard (e.g., "UDUNITS" 
      (the default) or "UCUM") that you (the ERDDAP admin) are using to 
      specify units.  The value is case-sensitive.
      This is used by ERDDAP's SOS server to determine if the units need to
      be converted to UCUM units for WMS and SOS GetCapabilities responses. 
      -->
      <units_standard>UDUNITS</units_standard>
      
      <!-- For the wms examples, pick one of your grid datasets that has
      longitude and latitude axes.
      The sample variable must be a variable in the sample grid dataset.
      The bounding box values are minx,miny,maxx,maxy.
      -->
      <wmsSampleDatasetID>erdBAssta5day</wmsSampleDatasetID>
      <wmsSampleVariable>sst</wmsSampleVariable>
      <!-- The bounding box values are 
         minLongitude,minLatitude,maxLongitude,maxLatitude.
         Longitude values within -180 to 180, or 0 to 360, are now okay. -->
      <wmsSampleBBox>0,-75,360,75</wmsSampleBBox>
      
    • REQUIRED: In setup.xml, copy and paste this new suggested <startHeadHtml> to replace your old version. But feel free to make changes for your preferences.
      <!-- startHeadHtml has the start of the HTML document and the 
      'head' tags (starting at "<!DOCTYPE>", but not including 
      "</head>") for all HTML web pages. 
      This may include &erddapUrl;, which is expanded to be 
        [baseUrl]/erddap (or [baseUttpsUrl]/erddap if the user is logged in).
      If your ERDDAP allows users to log in, all referenced image files, 
        css files, etc. must be in [tomcat]/content/erddap/images or a 
        subdirectory and must be referenced here with 
        &erddapUrl;/images/[fileName].
      
      favicon.ico is the image that browsers associate with your website.
      More more information, see https://en.wikipedia.org/wiki/Favicon .
      You can use your own favicon.ico file by putting it in 
        [tomcat]/content/erddap/images. 
      
      *** Optional: you can change the appearance of all of your 
      ERDDAP's HTML pages by changing the CSS <style> settings below.
      
      For an example of a very different style, change the import reference
      to <tomcat>/content/erddap/images/erddapAlt.css
      
      *** If your CSS style includes links to files (e.g., images), that 
      style information must be inline in the style tag below, after the
      'import' line, not in the .css file.  
      Put all of the (e.g., image) files in the 
      [tomcat]/content/erddap/images directory (or a subdirectory) and 
      reference them below starting with &erddapUrl;.
      Why? On ERDDAP https: web pages, *all* links should use "https:" 
      (not "http:"); otherwise, most browsers consider the web page not 
      fully secure.  Because ERDDAP would use the same .css file for 
      http: and https: web pages, the links within the .css file wouldn't 
      switch between http: and https:.  There doesn't seem to be a way 
      around this other than using inline style information.
      -->
      <startHeadHtml><![CDATA[ 
      <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
        "http://www.w3.org/TR/html4/loose.dtd">
      <html>
      <head>
      <title>ERDDAP</title>
      <link rel="shortcut icon" href="&erddapUrl;/images/favicon.ico">
      <style type="text/css">
      <!--
        @import "&erddapUrl;/images/erddap.css";
      -->
      </style>
      ]]></startHeadHtml>
      
      <!-- The tableCommonBGColor MUST be the same color as the 
         table.commonBGColor in erddap.css above. Suggested is #f1ecd8. 
         But if you use erddapAlt.css, change this to #e7dec5. -->
      <tableCommonBGColor>#f1ecd8</tableCommonBGColor>
      
      <!-- This is used, e.g., for the type=variable rows on the metadata
        info tables. -->
      <tableHighlightBGColor>#cceecc</tableHighlightBGColor>
      
      Thanks to POST, Hans Vedo, and Rick Blair.
    • REQUIRED: In setup.xml, in <startBodyHtml>, change the <body> tag to be just <body>, since the style is now set by erddap.css.
    • REQUIRED: In setup.xml, change to this <endBodyHtml> (but change the email address to your email address and feel free to make other changes):
      <!-- The end of the body of the HTML code for all HTML web pages
        (with "</body>" at the end). 
      This may include &erddapUrl;, which is expanded to be 
        [baseUrl]/erddap (or [baseUttpsUrl]/erddap if the user is logged in).
      If your ERDDAP allows users to log in, all referenced image files, etc. 
        must be in [tomcat]/content/erddap/images or a subdirectory
        and must be referenced here with &erddapUrl;/images/[fileName].
      
      You can change this, but please keep "ERDDAP, Version &erddapVersion;"
      and these references to the Disclaimers and Privacy Policy. -->
      <endBodyHtml><![CDATA[ 
      <br>&nbsp;
      <hr>
      ERDDAP, Version &erddapVersion;
      <br><font class="subduedColor">Questions, comments, 
        suggestions?  Please send an email to 
        <tt>bob dot simons at noaa dot gov</tt>
      <br>and include the ERDDAP URL directly related to your question
        or comment.
      <br>
        <a href="&erddapUrl;/legal.html">Disclaimers</a> | 
        <a href="&erddapUrl;/legal.html#privacyPolicy">Privacy 
          Policy</a>
      </font>
      </body>
      ]]></endBodyHtml>
      
    • HIGHLY RECOMMENDED: In setup.xml, the recommended <theShortDescriptionHtml> is now
      <theShortDescriptionHtml><![CDATA[ 
      <h1>ERDDAP</h1>
      This web site (the Environmental Research Division's Data Access 
      Program) aggregates scientific data from diverse local and remote 
      sources and offers you a simple, consistent way to download subsets 
      of the data in common file formats and make graphs and maps.
      This particular ERDDAP installation has oceanographic data
      (for example, data from satellites and buoys).
      
      [standardShortDescriptionHtml]
      ]]></theShortDescriptionHtml>
      
      Feel free to change this, particularly the last sentence in the first paragraph.
    • In setup.xml, emailEverythingTo and emailDailyReportTo can now be comma-separated lists of email addresses. The first emailEverythingTo is special, e.g., subscriptions to EDDXxxxFromErddap datasets use that email address. Thanks to John Maurer.
    • Email errors are now logged to the [bigParentDirectory]/logs/emailLogYYYY-MM-DD.txt file.
    • In setup.xml, there is a new, optional parameter to set email account properties (usually right after <emailPassword>):
        <emailProperties>propertyName1|propertyValue1|propertyName2| propertyValue2|...</emailProperties>
      For example, gmail accounts need
        <emailProperties>mail.smtp.starttls.enable|true</emailProperties>
      The default is nothing. Thanks to Rich Signell.
    • REQUIRED: If you use EDDTableCopy or EDDGridCopy, you must DELETE all [bigParentDirectory]/copy/ directories and files that contain "xh" in the directory or file names after stopping the old ERDDAP and before starting the new ERDDAP so those files will be re-copied. I'm very sorry, but it was important to make the change and hopefully it affects few admins and few files.
      In Linux, you can find these files with,
        cd [bigParentDirectory]/copy
        find . *xh*

      In Windows, you can find these files with,
        Start | Search
         What do you want to search for: Documents
          All or part of the file name: xh
          Look in: Browse -> [bigParentDirectory]/copy
          Click on 'Search'
           ^A to select them all
           Del to delete them all
    • REQUIRED: In datasets.xml, for EDDTableFromDatabase datasets, for date and timestamp variables, change the dataType to double and the units to seconds since 1970-01-01T00:00:00Z. We REQUIRE that you store timestamp data in the database *with* a timezone. Without timezone information, the queries that ERDDAP sends to the database and the results that ERDDAP gets from the database via JDBC are ambiguous and are likely to be wrong. We tried, but found no reliable way to deal with "timestamp without timezone" data. We think this is good practice anyway. After all, "timestamp without timezone" data has an implied timezone. While it is great that the time zone is obvious to the database admin, it makes sense to specify it explicitly so that other software can properly interact with your database. Thanks/sorry Michael Urzen.
    • HIGHLY RECOMMENDED: In datasets.xml, to enable .subset web pages for faceted search of your tabular datasets, you need to add <subsetVariables> to the dataset's global attributes.
    • RECOMMENDED: In datasets.xml, if you have the dataset with datasetID="pmelGtsppp", please change it to be
        <dataset type="EDDTableFromDapSequence" datasetID="pmelGtsppp" active="false">
      Whether or not you had that dataset, feel free to add this new GTSPP dataset:
        <dataset type="EDDTableFromErddap" datasetID="erdGtsppBest">
          <sourceUrl>https://coastwatch.pfeg.noaa.gov/erddap/tabledap/erdGtsppBest</sourceUrl>
        </dataset>
    • RECOMMENDED: In datasets.xml, there are new valid options for the <cdm_data_type> global attribute, so you should review/change the value for your datasets.
    • In datasets.xml, the new <sourceNeedsExpandedFP_EQ> is helpful if the source server doesn't consistently handle &variable=value tests correctly (because of the general difficulty of testing the equality of floating point numbers (external link)). sourceNeedsExpandedFP_EQ is set to true by default (the safest setting), so you don't need to make any changes.
    • New EDDTableFromAsciiFiles. Thanks to Jerry Yun Pan.
    • New EDDTableFromThreddsFiles. Thanks to Roy Mendelssohn.
    • Changes to EDDTableFromNcFiles lets it be used with a wider range of files.
    • EDDTableFromBMDE has been disabled. There are no longer any active, appropriate, data sources.
    • In GenerateDatasetXml, the new EDDGridFromThreddsCatalog harvests an entire THREDDS catalog (or a subset) and generates datasets.xml content. Thanks to the UAF project.
    • GenerateDatasetsXml and DasDds now also put their results in [bigParentDirectory]/logs/log.txt. Thanks to Rich Signell and Charles Carleton.
    • Many improvements to the login system. Thanks to POST.
  • Things ERDDAP Programmers Need to Know and Do:
    • There have been changes in the /WEB-INF/lib/ directory. Please change your javac and java classpath settings accordingly.
    • There is a new [yourUrl]/erddap/version service to determine the version of an ERDDAP. The response is text, e.g., ERDDAP_version=1.24 If you get an HTTP 404 Not-Found error message, treat the ERDDAP as version 1.22 or lower. Thanks to POST.
  • Small Changes and Bug Fixes:
    • EDDTableFromSos changes:
      • Dropped support for reading IOOS SOS XML responses.
      • Added support for reading IOOS SOS text/csv. (So NOS SOS servers currently aren't supported.)
      • Made lots of changes related to IOOS SOS server details.
      • Added support for BBOX queries for IOOS SOS and OOSTethys SOS servers.
      These changes result in a big speed up for relevant data requests. Thanks to IOOS SOS.
    • Text in .mat tabular data files is now saved correctly. Thanks to Roy Mendelssohn.
    • WMS
      • OpenLayers is now bundled with ERDDAP for use on the WMS web pages. This fixes the problem caused when OpenLayers changed a few months ago and prevents future problems.
      • In the WMS GetCapabilities response, the <OnlineResource> value is now the URL of the WMS service. Thanks to Charlton Galvarino.
      • A legend is displayed on the WMS web page to show the colorbar. Thanks to Emilio Mayorga.
    • EDDGridAggregateExistingDimension constructor had problems if an axis' sourceValues weren't equal to their destinationValues, e.g., if source time was something other than "seconds since 1970-01-01". Thanks to Todd Spindler.
    • In TableWriterGeoJson, the excess ',' after bbox[...] as been removed. Thanks to Greg Williams.
    • Many small changes and bug fixes.

     

Changes in ERDDAP version 1.22 (released 2009-07-05)

  • The SlideSorter bug introduced in 1.20 is fixed.
  • The OBIS bug introduced in 1.20 is fixed.
  • The references to Jason datasets on the images/gadgets/GoogleGadgets page were removed.
     

Changes in ERDDAP version 1.20 (released 2009-07-02)

  • ERDDAP administrators, please add this to your setup.xml file:
    <!-- If you want to restrict access to some datasets, you need to 
    specify the method used for logging on (authentication). See the info 
    at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security
    Currently, the options are: "" (logins not supported, the default),
    "custom", "openid". Note that openid login doesn't work when testing 
    with localhost (https://127.0.0.1:8443).
    -->
    <authentication></authentication>
    
    <!-- This specifies how you have stored passwords in the roles tags 
    in datasets.xml. If you aren't storing any passwords this is irrelevant.
    The options (in order of increasing security) are: "plaintext", "MD5", 
    or "UEPMD5" (MD5(UserName:ERDDAP:Password), the default).
    You should only use "plaintext" or "MD5" if you need to match values 
    stored that way in an external password database.  See the info at
    https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security
    -->
    <passwordEncoding>UEPMD5</passwordEncoding>
    
    <!-- This determines whether datasets that the user doesn't currently
    have access to (because he isn't logged in or because his roles don't
    allow access) should be shown on lists of data sets 
    (e.g., from full text search, categorize, view all datasets, ...).
    The options are: "true", or "false" (the default).
    If false, no information about the dataset (even its existence) is 
      shown to users who don't have access to it.
    If true, some information about the dataset (title, summary, etc) is
      shown to users who don't have access to it.  
      If the user clicks on a link to a dataset he doesn't have access to,
      he will get an error message and be prompted to log in.
    -->
    <listPrivateDatasets>false</listPrivateDatasets>
    
    <!-- If the number of requests between two runs of LoadDatasets 
    exceeds unusualActivity, an email is sent to emailEverythingTo.
    The default is 10000.
    -->
    <unusualActivity>10000</unusualActivity>
    
  • New dataset types EDDGridCopy and EDDTableCopy make and maintain a local copy of another EDDGrid or EDDTable dataset's data and serve data from the local copy. These are very easy to use and very effective solutions to some of the biggest problems with serving data from remote data sources:
    • Accessing data from a remote data source can be slow (for a variety of reasons).
    • The remote dataset is sometimes unavailable (again, for a variety of reasons).
    • Relying on one source for the data doesn't scale well (e.g., when many users and many ERDDAPs utilize it).
    Plus, the local copy is a backup of the original, which is useful in case something happens to the original.

    There is nothing new about making a local copy of a dataset. What is new here is that these classes make it *easy* to create and *maintain* a local copy of data from a *variety* of types of remote data sources and *add metadata* while copying the data.

    These dataset types are part of a complete set of features that simplify the creation of grids/clusters/federations of ERDDAPs to handle very heavy loads (e.g., in a data center).

  • New dataset type EDDTableFromDatabase gets data from a local or remote database table.
  • ERDDAP now has a security system that supports authentication (letting users log in) and authorization (granting them access to certain private datasets).
  • There are two, new, command-line tools to help ERDDAP administrators generate the XML for a new dataset in datasets.xml:
    • GenerateDatasetsXml can generate a rough draft of the dataset XML for almost any type of datasets.
    • DasDds helps you repeatedly test and refine the XML for a dataset.
    ERDDAP's GenerateDatasetsXml web pages have been removed. For security reasons, they only supported a few dataset types. The new command line tools are a better solution.
  • The new status page lets anyone (but especially administrators) view the status of an ERDDAP from any browser by going to [baseUrl]/erddap/status.html .
  • Tabledap now supports server-side functions:
    • &distinct() removes duplicate rows from the response table,
    • &orderBy(...) lets you specify how the response table should be sorted,
    • &orderByMax(...) lets you specify how the response table should be sorted and removes all rows except for the rows with the maximum values in the last specified column. This can be used, for example, to get the last available data for each station.
  • Tabular datasets can now include additional dateTime variables which aren't named "time". These variables are recognized by their "units" metadata, which must contain " since " (for numeric dateTimes) or "yy" or "YY" (for formatted String dateTimes). But please still use the destinationName "time" for the main dateTime variable.
  • ERDDAP now generates a sitemap.xml file, which tells search engines that your ERDDAP only needs to be crawled every month. ERDDAP administrators, please follow these instructions to notify the search engines about the new sitemap.xml file.
  • ERDDAP's error messages are now much shorter and geared to clients (not programmers). Thanks to Greg Williams.
  • <requestBlacklist> now also supports IP addresses where the last number has been replaced by *.
  • Requests for .json and .geoJson files may now include an optional jsonp (external link) request by adding "&.jsonp=functionName" to the end of the query. Basically, this just tells ERDDAP to add "functionName(" to the beginning of the response and ")" to the end of the response. If originally there was no query, leave off the "&" in your query. Thanks to Greg Williams.
  • Lots of new statistics were added to the Daily Report.
  • On web pages with lists of datasets, institution and id are now at the far right. This moves subscription and other more useful columns into view on narrow computer screens.
  • On all web pages, the page's title (based on the <title> in the <startHeadHtml> that you define in setup.xml) is modified to include a better description of the web page (for example, by including the current dataset's title and institution).
  • Xmx information is now included with the memory information printed in log.txt, the Daily Report, and on status.html. Thanks to Ellyn Montgomery.
  • Erddap has additional, general-purpose protection against all errors (e.g., OutOfMemoryError). Thanks to Charles Carleton.
  • Improvements to error handling if the response has already been committed.
  • Change: EDDTableFromFiles and EDDGridFromFiles now just allow <metadataFrom> first or last. penultimate is no longer supported. And first and last are now based on the files' lastModifiedTime.
  • Bug fix: in EDDTableFromSOS, invalid info for one station threw an exception and caused the whole dataset to be rejected. Now, those stations are just ignored (and the error message is logged to log.txt). Thanks to Rick Blair.
     

Changes in ERDDAP version 1.18 (released 2009-04-08)

  • Bug fix: Starting in 1.14, the EDDTable Data Access Form and Make A Graph web page didn't properly deal with quoted constraints.
  • Bug fix: Starting in 1.14, EDDTableFromDapSequence didn't handle time constraints correctly if the source time units weren't "seconds since 1970-01-01T00:00:00".
     

Changes in ERDDAP version 1.16 (released 2009-03-26)

  • ERDDAP administrators:
    • This is an important release because it fixes a bug that left an ERDDAP thread running if you used Tomcat Manager to Stop/Start or Reload ERDDAP. So when you install 1.16, don't just use Tomcat manager to undeploy the old ERDDAP and deploy the new ERDDAP. Instead: undeploy the old ERDDAP, restart Tomcat (or the server), then deploy the new ERDDAP. It's always a good idea to do that when installing a new version.
    • Please add <requestBlacklist></requestBlacklist> to your datasets.xml. This can be used to specify a list of client IP addresses to be blocked (e.g., to fend off a Denial of Service attack or an overly zealous web robot).
  • There is now a [bigParentDirectory]/logs directory to hold the ERDDAP log files. When you start ERDDAP, it makes an archive copy of the log.txt and log.txt.previous files with a time stamp. If there was trouble before the restart, it may be useful to analyze these files.
  • ERD's ERDDAP now has the subscription system turned on.
  • ERDDAP once again allows (but still doesn't recommend) the "%26" encoding of "&" in request URLs (see the related v1.14 change).
  • Several new additions to the Tally section of the Daily Report.
  • Small bug fixes in generateDatasetsXml.
  • A few small bug fixes.
     

Changes in ERDDAP version 1.14 (released 2009-03-17)

  • Changes for users:
    • In grid data requests, ERDDAP now supports: last-n where n is an integer number of indices and (last-d) where d is a numeric value (for time, it is in seconds).
    • In tabular data requests, String constraints now require double quotes around the value, for example, &id="NDBC40121" This is required by the DAP protocol.
    • In tabular data requests, ERDDAP now requires that all constraints be properly percent encoded. Browsers do this automatically, so this mostly affects computer programs/scripts that are accessing ERDDAP.
    • Previously, the embed a graph web page and the ERDDAP Google Gadget web page said to replace the "&" in the image's URL with "%26". From now on, you should replace the "&" in the image's URL with "&amp;". So you need to replace any "%26" in existing web pages and Google Gadgets with "&amp;". (Sorry)
  • ERDDAP administrators, please:
    • Add the following to your setup.xml file (and change the flagKeyKey value):
      <!-- ERDDAP has a service that lets remote users set a flag
      to notify ERDDAP to try to reload a dataset.
      These requests use a key which is generated based
      on baseUrl/warName, a datasetID, and flagKeyKey.
      *** Change this once, to any text (a favorite quote? random text? 
      It doesn't matter). Normally, you won't ever change this again.
      But if you you think someone is abusing the flag system,
      change this text again, restart ERDDAP, and send
      all of the users of the flag system the relevant new flagKeys
      (see the list in the Daily Report). -->
      <flagKeyKey>A stitch in time saves nine. CHANGE THIS!!!</flagKeyKey>
      
      <!-- ERDDAP has an email/URL subscription system which sends a user
      an email or pings a url whenever a dataset of interest changes.
      (This is different from the RSS system, which is always active.)
      The system relies on the server being able to send out 
      emails to people to validate their subscription requests.
      The emails appear to come from the emailFromAddress below.
      So if your server can't send out emails, don't make this system active.
      You may choose (for whatever reason) to make this system active or not, 
      so valid values below are "true" (the default) and "false".
      Note that if you change this and restart ERDDAP, the list of 
      subscriptions (in [bigParentDirectory]/subscriptionsV1.txt) isn't
      affected. See also the subscriptionEmailBlacklist in datasets.xml.
      -->
      <subscriptionSystemActive>true</subscriptionSystemActive>  
    • On the line after <emailUserName> in in setup.xml, add
      <emailPassword>myPassword</emailPassword> <!-- optional; if absent, emails can't be sent to non-local addresses >
      and enter your real password.
    • You can change <wmsSampleBBox> in setup.xml to include longitude values up to 360, e.g.,
      <!-- The bounding box values are 
         minLongitude,minLatitude,maxLongitude,maxLatitude.
         Longitude values within -180 to 180, or 0 to 360, are now okay. -->
      <wmsSampleBBox>0,-75,360,75</wmsSampleBBox>  
    • In your datasets.xml file, rename dataset type EDDTableFromNc4DFiles to EDDTableFromNcFiles (which now supports files with any number of dimensions). If you had an EDDTableFromNc4DFiles dataset:
      1. You MUST change to type="EDDTableFromNcFiles" in your datasets.XML file.
      2. You MUST add a <nDimensions>4</nDimensions> tag to the dataset's XML.
      3. You may add the new <sortFilesBySourceNames> tag to specify the internal order for the files, which determines the overall order of the data returned.
      For details, see EDDTableFromFiles.
    • In the past, for EDDTableFromDapSequence, for OPeNDAP DRDS servers, in datasets.xml, we used <sourceCanConstrainStringsRegex>~=</sourceCanConstrainStringRegex>. But we now see that the DRDS regex support is more limited than ERDDAP's, so we recommend <sourceCanConstrainStringsRegex></sourceCanConstrainStringRegex> so that regex constraints are not passed to the source, but are instead handled by ERDDAP.
    • Revamped handling of sourceCanConstrain... in datasets.xml by EDDTableFromDapSequence and (internally) all EDDTable dataset types. The new system is simpler and better reflects the variability of different data sources. You may need to modify the XML for your datasets in datasets.xml.
  • There are several new features which are useful by themselves, but when combined, also facilitate the creation of grids/clusters/federations of ERDDAPs.
    • New dataset types:
    • RunLoadDatasets and LoadDatasets were revamped so that ERDDAP is very responsive to reloading datasets based on files in the flag directory (often <5 seconds if main loadDatasets is currently done).
    • New service to allow a URL to create a flag file for a given dataset, e.g.,
      https://coastwatch.pfeg.noaa.gov/erddap/setDatasetFlag.txt?datasetID=rPmelTao&flagKey=123456789
      creates a flag file in the flag directory for rPmelTao (although the flagKey here is wrong).
    • New subscription service so that any client can specify an action which will be done when a specific dataset is created (when ERDDAP is restarted) and whenever the dataset changes in any way. This system can be disabled via <subscriptionSystemActive> in setup.xml. The ERDDAP Daily Report now lists all of the subscriptions and includes the URL needed to cancel each one, in case you feel the system is being abused. In datasets.xml, there is a new, optional <subscriptionsEmailBlacklist> tag so that administrators can specify a comma-separated list of email address which are immediately blacklisted from the subscription system.
    • New <onChange> attribute in datasets.xml lets the ERDDAP administrator specify an action which will be done when a specific dataset is created (when ERDDAP is restarted) and whenever the dataset changes in any way.
    • Improvements to full text search: storing the search string for each dataset now uses 1/2 the memory. The search algorithm (Boyer-Moore-like) is now 3X faster.
    • Emails from ERDDAP now always prepend the subject and content with [erddapUrl], so that it will be clear which ERDDAP this came from (in case you administer multiple ERDDAPs).
    • More extensive statistics gathering for the Daily Report email.
    • New log file [bigParentDirectory]/emailLogYEAR-MM-DD.txt logs all emails sent by ERDDAP each day. This is especially useful if your server can't actually send emails -- you can at least read them in the log.
    • ERDDAP now makes a [bigParentDirectory]/cache/(datasetID) directory for each dataset since there may be lots of files cached.
  • New RSS 2.01 feed for each dataset (look for the orange RSS icons on lists of datasets, Data Access Forms, and Make A Graph web pages).
  • EDDGrid .kml responses now use tiled images ("superoverlays" - dynamically generated quadtree images). The initial image loads into GoogleEarth much faster than before. The resolution of the map increases as you zoom in, up to the full resolution of the dataset. Recommend: users should request .kml for one time point, but the dataset's entire longitude,latitude range. Unfortunately, support for time ranges was removed (I hope it will come back).
  • ERDDAP now adds Expires and Cache-Control max-age headers (external link) to all files requested from the /images directory. This greatly reduces the number of static file requests sent to ERDDAP and thus greatly speeds up most ERDDAP page loads. Also, many JavaScript file references moved to the bottom of their HTML pages, which also speeds up many ERDDAP page loads. Thanks to the book "High Performance Web Sites" by Steve Souders and the ySlow addition to the FireBug plugin in FireFox.
  • ERDDAP switched from netcdf-java 2.2.22 to netcdf-java 4.0. Among other things, this allows EDDGridFromNcFiles to read HDF .hdf, as well as GRIB .grb and NetCDF .nc files.
  • EDDGridFromDap and EDDGridFromNcFiles now also support DArray (as well as DGrid) dataVariables. If a dimension doesn't have a corresponding coordinate variables, ERDDAP creates an axis variable with the index values (e.g., 0, 1, 2, ..., 311, 312). So all other aspects of EDDGrid remain the same:
    * It still serves all datasets as Grids, with an axis variable for each dimension.
    * Queries can still request values from the axis variables.
    Thanks to Charles Carleton, Thomas Im, Dorian Raymer, and others.
  • The WMS OpenLayers pages now have a default longitude,latitude range that is a little larger than the dataset's range (not the exact range, so the context of small datasets is more obvious). The default range may now also be 0 to 360, which allows the full range of many datasets to be shown now. Thanks to Todd Spindler.
  • New sliders on some Data Access Forms and Make A Graph web pages. They simplify (crude) specification of the desired data and offer good visual feedback.
  • A new option for the <dataset> tags in datasets.xml: active="false".
  • References to ERD's ERDDAP changed from coastwatch.pfel (still works via proxy) to coastwatch.pfeg (preferred).
  • New support for data_min and data_max variable metadata attributes.
  • A partial solution to the WaitThenTryAgain / Partial Results Exception: Now, some requests that previously failed when a data source change was detected will succeed because ERDDAP will reload the dataset and re-request the data automatically, all in the context of the original request.
  • Bug fix: generateDatasetsXml was disabled in ERDDAP version 1.12. Thanks to Ellyn Montgomery for pointing this out.
  • Small changes to error handling.
  • Many improvements to avoid/deal with possible race conditions (i.e., possible problems arising from the multi-threaded nature of ERDDAP) which caused small, infrequent problems.
  • Now, if an error message is written on an image, the image will only stay in the cache for ~5-10 minutes (not 60). Thanks to Cara Wilson.
  • The standard message when there is no data is now "Your query produced no matching results.", which is shorter, more accurate, and matches OPeNDAP servers.
  • EDDGrid no longer allows tied axis values.
  • Small changes to .ver and .help requests.
  • Many small changes and bug fixes.
     

Changes in ERDDAP version 1.12 (released 2008-10-31)

  • EDDTableFromSOS once again works with NDBC SOS and works with the new NOS SOS.
  • EDDTableFromBMDE now requires ERDDAP admin to specify dataVariables.
  • EDDGrid no longer requires that lat and lon be evenly spaced for .transparentPng or .kml. Thanks to Todd Spindler.
  • A few small changes.
     

Changes in ERDDAP version 1.10 (released 2008-10-14)

  • New "colorBar" metadata for data variables in datasets.xml defines the default color bar settings for graphs and maps. See more information. This is important because it greatly improves the appearance of the default graphs and maps produced by Make A Graph and because the default graphs and maps now have a consistent color bar even when the client changes the requested time or geographic range. Also, this was necessary for WMS.
  • ERDDAP now serves most grid data via a WMS service. This is important because it shows that, in addition to getting data from many types of data servers, ERDDAP can distribute data via different protocols (DAP, WMS, ... more in future). See the client documentation. Or the documentation for administrators. Or try it out.
  • New support for longitude values >180 in .kml files.
  • New cdm_data_type: Other .
  • ERDDAP now supports "boolean" source dataType. See more information This will become useful for the future EDDTableFromDatabase.
  • New EDDTableFromBMDE supports DiGIR/BMDE data sources.
  • EDVGridAxis now allows descending sorted values. The pmelOscar datasets needed this.
  • ERDDAP now returns HTTP errors (e.g., "404 for resource/page not found") in more situations, instead of HTML pages with error messages.
  • Lots of changes/additions to the ERDDAP documentation.
  • Lots of small changes.
  • Some bug fixes.
  • Things ERDDAP administrators should do to upgrade to this version:
    • In datasets.xml, for any EDDTableFromSOS datasets, change "observedProperty" metadata to "sourceObservedProperty".
    • The rules for an axisVariable or dataVariable's destinationName are now stricter. You need to check that your variable names are valid. Either check them by hand, or run ERDDAP and look at the error messages in the report that is emailed to the administrator.
    • In datasets.xml, if you want a grid data variable to be accessible via WMS, you need to add colorBar metadata. At least, for example,
        <att name="colorBarMinimum" type="double">0</att>
        <att name="colorBarMaximum" type="double">32</att>

      See more information.
    • Add the following to your setup.xml (but customize with your information):
      
      <!-- drawLand specifies the default Make A Graph setting for 
      whether the landmask should be drawn "over" (the default) or "under" 
      surface data on maps. "over" is recommended for primarily 
      oceanographic data (so that grid data over land is obscured by the 
      landmask). "under" is recommended for all other data.
      -->
      <drawLand>over</drawLand>  
      
      <!-- Information about the ERDDAP administrator is used for the 
      SOS and WMS servers. You MUST CHANGE these to describe your 
      installation. 
      -->
      <adminInstitution>NOAA Environmental Research 
      Division</adminInstitution>
      <adminIndividualName>Your Name</adminIndividualName>
      <adminPosition>Webmaster</adminPosition>
      <adminPhone>your-phone-number</adminPhone>
      <adminAddress>99 Pacific St, Suite 255A</adminAddress>
      <adminCity>Monterey</adminCity>
      <adminStateOrProvince>CA</adminStateOrProvince>
      <adminPostalCode>93940</adminPostalCode>
      <adminCountry>USA</adminCountry>
      <adminEmail>yourName@yourSite</adminEmail>
      
      <!-- Information about the ERDDAP administrator is used for ERDDAP's
      SOS server. You MUST CHANGE these to describe your installation. 
      -->
      <sosTitle>NOAA Environmental Research Division SOS</sosTitle>
      <sosAbstract>NOAA Environmental Research Division's ERDDAP makes 
        data from multiple sources available via the SOS 
        protocol.</sosAbstract>
      <sosKeywords>Weather, Ocean Currents, Temperature, 
        Salinity</sosKeywords>
      <sosAccessConstraints>NONE</sosAccessConstraints>
      <sosFees>NONE</sosFees>
      
      <!-- Information about the ERDDAP administrator is used for 
      ERDDAP's WMS server. You MUST CHANGE these to describe your 
      installation. -->
      <wmsTitle>NOAA Environmental Research Division 
      WMS</wmsTitle>
      <wmsAbstract>NOAA Environmental Research Division's ERDDAP makes
      data from multiple sources available via the WMS 
      protocol.</wmsAbstract>
      <wmsKeywords>Weather, Ocean Currents, Temperature, 
        Salinity</wmsKeywords> 
      <wmsAccessConstraints>NONE</wmsAccessConstraints>
      <wmsFees>NONE</wmsFees>
      <!-- For the wms examples, pick one of your grid datasets that has 
      longitude and latitude axes. The sample variable must be a variable 
      in the sample grid dataset.  The bounding box values are 
      minx,miny,maxx,maxy.
      -->
      <wmsSampleDatasetID>erdBAssta5day</wmsSampleDatasetID>
      <wmsSampleVariable>sst</wmsSampleVariable>
      <wmsSampleBBox>0,-75,180,75</wmsSampleBBox>
      
      

Changes in ERDDAP version 1.08 (released 2008-07-13)

  • A new web service in ERDDAP, generateDatasetsXml, assists ERDDAP administrators by creating a rough draft of the XML needed to describe a dataset in datasets.xml
  • Some changes/bug fixes related to allowing griddap to be seen by netcdf-java as an opendap server, including: global metadata is now labeled "NC_GLOBAL" (instead of "GLOBAL").
  • The EDDGrid and EDDTable Data Access Forms now utilize query information in the URL. So, for example, if a user goes from a Make A Graph form to a Data Access Form, the constraints are now properly transferred.
  • tabledap's Make A Graph now allows constraints on String variables.
  • EDDTable's Make A Graph now allows NaN constraints. Thanks to Steve Hankin.
  • Bug fix: EDDTable saveAsImage didn't properly recognize the .colorbar min and max values. Thanks to Steve Hankin
  • Many improvements to setupDatasetsXml. Thanks to Ellyn Montgomery.
  • Griddap requests now allow ()-style requests slightly outside of the actual axis range. This is appropriate since ()-values are rounded to the nearest actual value. Thanks to Cindy Bessey
  • I made the FloatArray and DoubleArray test of isEvenlySpaced more sophisticated. It will always be imperfect (because the test would need to be customized for each dataset), but it should be better. Thanks to Ellyn Montgomery.
  • I moved setup.html and setupDatasetsXml.html erddap's /download directory and hard coded all links to them. Now, I can make changes and update the setup information immediately.
  • Many small changes. A few small bug fixes.
  • Things ERDDAP administrators should do to upgrade to this version:
    • Move <theShortDescriptionHtml> from your messages.xml to your setup.xml. It specifies the text that appears in the middle of the left side of the ERDDAP home page. Also, add <h1>ERDDAP</h1> (or some other headline) to the top of it. Or, copy <theShortDescriptionHtml> in the new setup.xml (from the new erddapContent.zip) into your setup.xml.
       

Changes in ERDDAP version 1.06 (released 2008-06-20)

  • New support for IOOS DIF SOS data sources.
  • Many small changes. A few small bug fixes.
     

Changes in ERDDAP version 1.04 (released 2008-06-10)

  • New Slide Sorter feature.
  • New Google Gadgets page and examples.
  • Bug fix in EDDGrid.saveAsNc for variable with scale and addOffset.
     

Changes in ERDDAP version 1.02 (released 2008-05-26)

  • New EDDGridSideBySide allows for different axisVariables[0] sourceValues.
  • All of the currents and winds datasets were merged into EDDGridSideBySide datasets.
  • Images from image requests are now cached for 1 hour.
     

Changes in ERDDAP version 1.00 (released 2008-05-06)

  • Make A Graph web pages and graphics commands in URLs.
  • Support for flag files to force reloading a dataset.
  • New dataset type: EDDTableFrom4DFiles (the first subclass of EDDTableFromFiles).

 

Contact

Questions, comments, suggestions? Please send an email to bob dot simons at noaa dot gov and include the ERDDAP URL directly related to your question or comment.
 

ERDDAP, Version 1.74
Disclaimers | Privacy Policy