NOAA   ERDDAP
Easier access to scientific data

Brought to you by NOAA NMFS SWFSC ERD    
 

Changes

Here are the changes associated with each ERDDAP release.

Changes in version 1.42 (released 2012-11-26)

  • New Features:
    • (No major new features.)
  • Things ERDDAP Administrators Need to Know and Do:
    • If you are upgrading from ERDDAP 1.38 or 1.40, there were no changes that require you to make changes to your configuration files (but you must use the new messages.xml file).
    • ERDDAP once again can run with Java 1.6. (ERDDAP v1.40 required Java 1.7.) We still strongly recommend using the latest version of Java 1.7.
    • A new dataset type, EDDTableFromAwsXmlFiles, can read data from a set of Automatic Weather Station (AWS) XML data files. Thanks to Lynn Dewitt and the Exploratorium.
  • Small changes/Bug Fixes:
    • Adjusted to changes to the NDBC SOS source data servers.
    • Adjusted to changes to the NOS COOPS ASCII services.
    • Made several small changes and bug fixes.

Changes in version 1.40 (released 2012-10-25)

  • New Features:
    • There is a new output file format for tabledap datasets: .ncCFMA, which saves the requested data in a .nc file which conforms to the CF Discrete Sampling Geometries Multidimensional Array options, and which therefore conforms to the NODC templates for storing this type of data. Thanks to NODC.
    • tabledap requests can now include time constraints such as &time>now-5days. See the documentation. Thanks to James Gosling.
  • Things ERDDAP Administrators Need to Know and Do:
    • If you are upgrading from ERDDAP 1.38, there were no changes that require you to make changes to your configuration files (but you must use the new messages.xml file).
    • ERDDAP public releases and internal milestones are available via ERDDAP on GitHub. For more information, see the Wiki for the ERDDAP project as well as the more general ERDDAP Programmer's Guide. (This was announced separately a few weeks after the ERDDAP 1.38 release.)
    • GenerateDatasetsXml has been improved.
      • The script was revised so it should work correctly all all Linux computers (not just a few).
      • It now adds creator_name, creator_email, and creator_url whenever possible.
      • Many other small improvements.
    • Refined how ERDDAP deals with time.
      • Internally, ERDDAP now handles times at millisecond precision (not seconds).
      • You can now optionally specify the time precision for a given dataset, see time_precision. For example, you might set a dataset to display time values with date precision (e.g., 1970-01-01).
      • Your current datasets will use the default settings, so they are unaffected by these changes and will continue to display time with seconds precision.
      Thanks to Servet Cizmeli and Philip Goldstein.
    • EDDTableFromNcCFFiles is a new dataset type which you can use in your datasets.xml file. It can read data from any of the numerous file formats defined by the CF Discrete Sampling Geometries conventions. Thanks to NODC and special thanks to Kyle Wilcox for making sample files for the huge number of valid DSG file formats and for making them publicly available.
  • Small changes/Bug Fixes:
    • Expanded the quickRestart system to all relevant EDDGrid and EDDTable subclasses.
    • Improved documentation, especially related to how to use griddap and tabledap from various client software.
    • Changed advanced search to support minTime and/or maxTime expressed as epochSeconds. Thanks to Lynn Dewitt.
    • Changed .htmlTable output to display urls and email addresses as links.
    • Added "rel=" and "rev=" to relevant <a href> tags. Thanks to Pat Cappelaere from the OGC REST project.
    • Improved protection against unrealistically large data requests, notably within tabledap, where it is a harder problem.
    • Moved more messages to messages.xml.
    • Made speed improvements.
    • Fixed EDDGridFromFiles to allow descending sorted axes. Thanks to Maricel Etchegaray.
    • Removed references to iGoogle since it will be discontinued.
    • Made several small changes and bug fixes.

Changes in version 1.38 (released 2012-04-21)

  • New Features:
    • ISO 19115 and FGDC - ERDDAP can automatically generate ISO 19115 and FGDC XML metadata files for each dataset. Links to the files are visible on every list of datasets (e.g., from Full Text Search) and also in Web Accessible Folders (WAF) (see the FGDC WAF and ISO 19115 WAF). Thanks to Ted Habermann, Dave Neufeld, and many others.
    • Full Text Searches for Datasets now support -excludedWord and -"excluded phrase" . Thanks to Rich Signell.
    • Searches for datasets now return results a page at a time. The default uses the parameter string: page=1&itemsPerPage=1000, but you can change the values in the URL of your request. Thanks to Steve Hankin and the UAF project.
    • OpenSearch - ERDDAP now supports the OpenSearch 1.1 standard for searching for datasets. Among other things, this allows catalog aggregation websites to do distributed searches (passing a search request to each catalog that it knows about).
    • Comma Separated Value (CSV) Files - ERDDAP now generates CSV files with just a comma between values (which Excel prefers), instead of comma+space. Thanks to Jeff deLaBeaujardiere.
    • Million Datasets - Several changes were made to support ERDDAPs having a huge number of datasets, perhaps even a million. Thanks to Steve Hankin and the UAF project.
  • Things ERDDAP Administrators Need to Know and Do:
    • A quick restart system allows ERDDAP to restart much faster.
      Please add this to your setup.xml file right after </datasetsRegex>:
            <!-- If true, when you start up ERDDAP, some types of datasets (e.g., 
            EDDGridFromDap) will used cached information (.dds, .das, etc.) to reload
            very quickly, without contacting the remote server.  The dataset's age 
            will be based on when the dataset was reloaded last.  Normally this 
            should be true (the default), but set it to false if you want to bypass 
            the cached information.
            <quickRestart>true</quickRestart>
    • Full text searches for datasets can now be done with the Lucene search engine (although we recommend the original search engine if you have fewer than 10,000 datasets) or the original search system.
      Please add this to your setup.xml file right after </displayDiagnosticInfo>:
            <!-- ERDDAP lets you choose between two search engines for full text searches:
            * original (the default) - is the best choice if your ERDDAP has fewer 
              than about 10,000 datasets.  It is very robust and trouble free. 
            * lucene - is the best choice for more than about 10,000 datasets.
              The advantages are that with any number of datasets it works fast 
              and uses very little memory.
              But there are many things that might go wrong with individual 
              queries and with the whole system. 
              And although its behaviour (the datasets it finds and the order that
              it ranks them) is almost identical to the original search engine,
              it has a few quirky, subtle, small differences.
            -->
            <searchEngine>original</searchEngine>
    • In setup.xml, you can/should now add two new categories to the comma-separated list of <categoryAttributes>:
      • global:keywords (add it right after global:institution) -- a new special case that parses a comma-separated list of keywords from the global keywords attribute to make a separate entry for each keyword.
      • variableName (add it at the end) -- a new special case that categorizes each of the dataVariable destinationNames.
    • In setup.xml, you can (but why?) tell ERDDAP not to offer FGDC and/or ISO 19115 metadata for any dataset by including
      <fgdcActive>false</fgdcActive>
      <iso19115Active>false</iso19115Active>

      The default values for these settings is true.
    • In datasets.xml, please consider improving the metadata for your datasets. ERDDAP now automatically generates ISO 19115 and FGDC XML metadata files for each dataset based on the dataset's metadata.
      So, good dataset metadata leads to good ERDDAP-generated ISO 19115 and FGDC metadata.
      See the new documentation for the many new RECOMMENDED Global Attributes.
    • In datasets.xml, if you want to tell ERDDAP to use a pre-made FGDC and/or ISO 19115 file that is somewhere on the server's file system instead of having ERDDAP generate these files, use:
      <fgdcFile>fullFileName</fgdcFile>
      <iso19115File>fullFileName</iso19115File>

      If fullFileName="" or the file isn't found, the dataset will have no FGDC and/or ISO 19115 metadata. So this is also useful if you want to suppress the FGDC and/or ISO 19115 metadata for a specific dataset.
    • In datasets.xml, for all EDDGridSideBySide and EDDGridAggregateExistingDimension datasets, make certain that child datasets have different datasetIDs than their parent datasets and than the other children. (For example, you could follow George Foreman's simple but effective system for naming his children.) If any names in a family are exactly the same, the dataset will fail to load (with the error message that the values of the aggregated axis are not in sorted order).
    • In datasets.xml, there were some changes to the list of valid ioos_category metadata values:
      • "pCO2" was changed to "CO2".
      • "Physical Oceanography" was added.
      • "Soils" was added.
    • In datasets.xml, ERDDAP no longer allows '.' in a datasetID. It was allowed but discouraged. (Sorry)
    • In datasets.xml, the setup for EDDTableFromThreddsFiles and EDDTableFromHyraxFiles has changed slightly because both classes were just rewritten to be more efficient (both classes now always make a local copy of all of the remote data files). See the documentation for setting up these classes: EDDTableFromHyraxFiles and EDDTableFromThreddsFiles. In particular, see the revised comments about <fileDir> (now irrelevant) and <sourceUrl> (now essential). Also, you should never wrap this class in EDDTableCopy for efficiency.
    • In datasets.xml, if you use EDDTableFromDatabase with an Oracle database, you should include a connectionProperty such as
      <connectionProperty name="defaultRowPrefetch">4096</connectionProperty>
      to specify how many rows of data to fetch at one time because the default is 10, which is horribly inefficient. See the Oracle documentation. MySql and PostgreSQL seem to have better defaults for this setting. Thanks to Kevin O'Brien.
    • If you use EDDTableFromDatabase, see the improved "Speed" documentation for additional suggestions to improve performance. Thanks to Kevin O'Brien.
    • In datasets.xml, for all EDDTable... datasets, in the Conventions and Metadata_Conventions global attributes, please refer to CF-1.6 (not CF-1.0, 1.1, 1.2, 1.3, 1.4, or 1.5), since CF-1.6 is the first version to include the changes related to the Discrete Sampling Geometry.
    • Programmers that are compiling the ERDDAP code need to add lib/lucene-core.jar to the list of jar files in their javac and java command line paths.
    • ERDDAP has a new service to convert a CF Standard Name to/from a GCMD Science Keyword. You may find this useful when generating global keywords metadata for the datasets in your ERDDAP.
    • Dealing with Bots - Please read this advice to prevent bots from crawling your ERDDAP in a stupid way.
    • Translation - The text on ERDDAP's web pages is now mostly in messages.xml and so suitable for translation to different languages (e.g., German, French). The messages now often use MessageFormat for formatting, also to aid in making translations. If you are interested in doing a translation, please email bob dot simons at noaa dot gov .
    • Sample datasets.xml - There were several small but significant errors in the sample datasets.xml. If you use those datasets, please get the newer versions from the new sample datasets.xml in the new erddapContent.zip. Thanks to James Wilkinson.
    • Git - I will try hard to make ERDDAP a GitHub project ASAP after this release.
  • Small changes/Bug Fixes:
    • A new palette, OceanDepth, is useful for depth values (positive is down), e.g., 0 (shallow) to 8000 (deep).
    • The .kml output from tabledap uses a better marker icon (it isn't fuzzy). And hovering over a marker now makes it bigger.
    • EDDTableFromFiles - In the last upgrade, the new netcdf-java library had tighter restrictions for variable names in .nc files. That caused problems for EDDTableFromFiles if a variable's sourceName had certain punctuation characters. EDDTableFromFiles is now modified to avoid that problem. Thanks to Thomas Holcomb.
    • The .subset page now supports 0/10/100/1000/10000/100000 instead of a check box for Related Data. The tooltip warns that 100000 may cause your browser to crash. Thanks to Annette DesRochers, Richard (Abe) Coughlin, and the IOOS Biological Project.
    • .../erddap/info/datasetID/index.html web pages now show urls and email addresses as clickable links. Thanks to Richard (Abe) Coughlin and the IOOS Biological Project.
    • Bug fix: In tabledap, for datasets with altitudeMetersPerSourceUnit < 0, queries with altitude constraints were handled incorrectly. Thanks to Kyle Wilcox.
    • Bug fix: EDDGridAggregateFromExistingDimension now supports more diverse TDS URLs. Thanks to ?

Changes in version 1.36 (released 2011-08-01)

Changes in version 1.34 (released 2011-06-15)

  • Changes:
    • Bug fix: Fixed memory leak that occurred on some 64-bit Java installations.
    • Bug fix: ERDDAP now correctly sets these global attributes when the latitude dimension's values ranged from high to low: geospatial_lat_min, geospatial_lat_max, Southernmost_Northing, Northernmost_Northing.

      Note that actual_range is unchanged: it may have low,high values or high,low values, since it is intended to indicate the range and the order of storage.

    • Small changes.
    • ERDDAP administrators don't need to make any changes to their setup.xml or datasets.xml.

Changes in version 1.32 (released 2011-05-20)

  • Changes:
    • Support for the newly ratified, CF Discrete Sampling Geometries (which unfortunately is not yet available online), which replaces the proposed CF Point Observation Conventions.
      ERDDAP users will see that cdm_feature_type=Station is replaced by TimeSeries and there are small changes to the files created for the .ncCF file type (flat_dimension is now called sample_dimension).
      ERDDAP administrators will need to make these changes in datasets.xml:
      • cdm_data_type=Station should be changed to cdm_data_type=TimeSeries.
      • cdm_data_type=StationProfile should be changed to cdm_data_type=TimeSeriesProfile.
      • cdm_station_variables should be changed to cdm_timeseries_variables.
      • cf_role=station_id should be changed to cf_role=timeseries_id.
    • New ioos_category options: "Colored Dissolved Organic Matter", "pCO2", "Stream Flow", "Total Suspended Matter".
    • Possible solution to a possible memory leak on 64-bit Java. [It didn't work.]
    • Small changes.

Changes in version 1.30 (released 2011-04-29)

  • New Features:
    • Support for 64-bit Java. When used with a 64-bit Java, ERDDAP can now use much more heap memory and handle many more simultaneous requests.
    • Support for .nc file requests up to 2GB (even without 64-bit Java) via better use of ERDDAP's handling of data in chunks.
    • Many 2X speed improvements in the code and 2X speed ups from Java 1.6 make ERDDAP 2X to 4X faster than before.
    • Memory saving improvements significantly lower ERDDAP's base memory usage.
    • For tabular datasets, ERDDAP is now fully aware of a dataset's cdm_data_type, and how the data maps to the CDM type. Currently, see the Microsoft Word file referenced at the top of proposed CF Point Observation Conventions. Perhaps someday soon, that Word file will be converted to .html and replace the current "OBSOLETE" information on that web page. Thanks to the NOAA UAF project.
    • For most EDDTable datasets, a new output file type option, .ncCF, creates .nc files which conform to the latest version of the proposed CF Point Observation Conventions (in the Microsoft Word file). These files are structured to reflect the CDM data type of the dataset. Since the proposed conventions just changed, as of this writing, the netcdf-java library does not yet support reading the file formats created by ERDDAP and interpreting them as CDM data files. It probably will soon. Thanks to the NOAA UAF project.
    • The View : Distinct Data option on the .subset web page is now a drop-down list that lets users specify the maximum number of rows of distinct data to be viewed (default = 1000). This change, and others, allow ERDDAP to work with datasets that have very large numbers of rows of distinct data. (The number of unique values for any single variable is still an issue, but it can be pretty high (20,000?) before the .subset and other web pages load really slowly.) Thanks to the NOAA UAF project.
    • .subset web pages have a new option: View Distinct Data Counts. Thanks to the GTOPP project.
    • To aid users, the distinct values (e.g., station names) are now shown on the Make-A-Graph and Data Access Forms. Thanks to the NOAA UAF project.
    • .transparentPng requests now support all types of graphs and data representations. It draws just the data -- no axes, legends, landmask, or anything else. This makes it possible to make images as layers of transparentPngs. If &.size=width|height is specified in the query (recommended), it is honored. The default is 360x360 pixels. The only exception is EDDGrid &.draw=surface, where the default (as before) is an image with ~1/pixel per datam (up to 3000 x and y pixels). Thanks to Fred Hochstaedter.
    • The WMS web pages now show the color bar for the dataset's variable(s). Thanks to Emilio Mayorga and others.
  • Things ERDDAP Administrators Need to Know and Do:
    • This release involves a lot of changes. They are all important. Please be patient and work through all of the changes listed below.
    • This version is being pushed out earlier than intended to deal with some Java security bugs. Unfortunately, several features/fixes intended for this ERDDAP version are not in this version. Sorry. Hopefully, the next version will be relatively soon (and much easier to upgrade to).
    • To avoid several security bugs in Java 6 update 23 and below, download and install the latest version of Java (Java 6 update 24 or higher). If you have a 64-bit operating system, please get a 64-bit version of Java.
    • If you are using Tomcat 5, you MUST upgrade to Tomcat 6 or 7 (preferred). If you are using Tomcat 6, consider upgrading to Tomcat version 7.
    • Please follow all of the instructions for setting up a new ERDDAP, but where relevant, you will be copying files from your old installation to the new installation, notably the [tomcat]/content/erddap directory and files. As part of that, note the new Tomcat setup recommendations.
    • The default erddap.css is now included in the erddap.war file.
      • To use the default erddap.css, delete your old [tomcat]/content/erddap/images/erddap.css .
      • If you modified [tomcat]/content/erddap/images/erddap.css, and want to keep using it: just leave it in place and replace the <input> section with:
        /* Small input items let more be shown on one screen
        (esp. Chrome and Safari). Google Chrome and Safari have
        default margin 2px, while others are 0. This sets all to 0.
        .skinny is used e.g., for the buttons above the image on
        a Make A Graph page. */
        input[type=button], input[type=submit], button {
          margin:0px; padding:0px 3px; }
        input[type=checkbox], input[type=password],
          input[type=text], select, textarea {
          margin:0px; padding:0px; }
        input[type=radio] {margin:0px 2px; padding:0px; }
        input.skinny {padding:0px 1px; }
    • In your [tomcat]/content/erddap/setup.xml:
      • Replace the comments and tags related to <partialRequestMaxBytes> and <partialRequestMaxCells> with
        <!-- When possible (and it isn't always possible),
        ERDDAP breaks source data requests into chunks to
        conserve memory. See the description of these tags in
        messages.xml. You can override the default chunk sizes
        here with
        For grids:
         <partialRequestMaxBytes>100000000</partialRequestMaxBytes>
        For tables:
         <partialRequestMaxCells>100000</partialRequestMaxCells>
        -->
      • Replace the comments related to <categoryAttributes> and consider modifying the tag's value:
        <!-- This is the comma-separated list (recommended:
        in alphabetical order) of the global attribute and
        variable attribute names which will be used to
        categorize the datasets and shown to clients at urls
        like .../erddap/categorize/ioos_category/index.html
        (ioos_category is unusual, but is used at ERD).
        If an attribute is a global attribute, identify it by
        prefixing it with "global:".
        -->
        <categoryAttributes>global:institution, ioos_category,
        long_name, standard_name</categoryAttributes>

         
        Individual <categoryAttributes> that are global attributes now MUST be identified via the prefix global: (e.g., global:institution). Others attributes are assumed to be variable attributes (e.g., standard_name). Also, institution values (the only ones) were left in original case. Now all category values are converted to lower case.
    • In your [tomcat]/content/erddap/datasets.xml:
      • Big change: ERDDAP has new requirements related to a tabular dataset's cdm_data_type. Notably, each dataset MUST have the correct metadata and variables related to the cdm_data_type. If not, the dataset won't load and will throw an error. See the documentation for cdm_data_type.
      • FYI: There is a new dataset type: EDDTableFromAsciiServiceNOS.
      • FYI: There are three newly allowed ioos_category options: Hydrology, Quality (e.g., for quality flags), and Statistics (e.g., mean).
      • For EDDTableFrom...Files datasets, remove any <nDimensions> tags. They are no longer needed or used.
      • For variables with destinationName=altitude, ERDDAP no longer forces the long_name to be Altitude. Please go through your datasets.xml and repeatedly search for <destinationName>altitude and add to that variable's <addAttributes>:
          <att name="long_name">Altitude</att>
        (or a slightly different long_name in special cases).
      • Optional: All EDDTableFromFiles subclasses support variable sourceName=global:... to convert global metadata from each file into a data variable. Thanks to Lynn DeWitt.
    • EDDTableFromDatabase users - ERDDAP comes with a new JDBC 4 driver for Postgres. For other databases, check the web for the lastest JDBC .jar file for your database. Since ERDDAP now uses Java 1.6+, JDBC 4 (not 3) is probably recommended.
    • FYI
      • EDDGridFrom...Files and EDDTableFrom...Files datasets now store the fileTable information in
        [bigParentDirectory]/datasetInfo/[datasetID]/*.nc files.
        Also, EDDTable datasets now store the subset information in
        [bigParentDirectory]/datasetInfo/[datasetID]/*.nc files. These files used to be
        [bigParentDirectory]/datasetInfo/[datasetID].*.json files.
        The old files will be deleted automatically when ERDDAP starts up. Or, you can delete all files (but leave the empty subdirectories) in [bigParentDirectory]/datasetInfo/.
      • I worked on a new EDDTableFromNcCFFiles which would read data from local and remote files using the proposed, new CF Point Observation Conventions. But it isn't in this release. There are problems in the netcdf-java libraries related to some methods for reading these files. And there were some very recent changes to the proposed CF Point Observation Conventions. When the netcdf-java library is fixed and updated to the latest proposal, I will resume work on this.
      • Running ERDDAP on Windows may have problems: notably, you may see in the [bigParentDirectory/logs/log.txt file that ERDDAP is sometimes unable to delete and/or rename files quickly. This is due to antivirus software (e.g., from MacAfee and Norton) which is checking the files for viruses. If you run into this problem (which can be seen by error messages in the log.txt file like "Unable to delete ..."), changing the antivirus software's settings may partially alleviate the problem.
        If the ERDDAP in Windows is just a test running on your desktop, this is just an annoyance.
        If the ERDDAP in Windows is your public ERDDAP, consider switching to a Linux server.
    • Slow First Startup - The first time you run ERDDAP after upgrading, ERDDAP may be slow to load the datasets. The way ERDDAP stores information about aggregated files has changed, so ERDDAP will need to re-read some info from all of those files. That will take time.
    • Errors on Startup - Given the changes related to cdm_data_type, it is likely that some of your datasets won't load and will throw errors. Carefully read the Daily Report email that ERDDAP sends you when ERDDAP is finished starting up. It will have a list of datasets that didn't load (at the top) and the reason they didn't load (near the bottom).
    • If you get stuck or have other questions, email the details to me: bob.simons at noaa.gov.
    • Programmers - If you write Java programs that run ERDDAP code, you need to change some of the command line parameter references:
      • Change joda-time-1.6.2.jar to joda-time.jar
      • Change the Postgres JDBC .jar reference to postgresql.jdbc.jar
  • Small Changes and Bug Fixes:
    • Improved connection handling to avoid hung threads.
    • Improved concurrency practices to handle nearly simultaneous identical requests more efficiently.
    • ERDDAP now uses netcdfAll-4.2.jar (renamed to netcdfAll-latest.jar). This switch necessitated several internal changes and causes a few small external changes, e.g., changes to how grib files are read and tiny changes to the .ncHeader output.
    • New feature: [erddap]/convert/fipscounty.html converts FIPS county codes to/from county names.
    • On maps, state boundaries are now dark violet, so they stand out a better on all background colors.
    • Tabular .kml output again uses a circular icon to mark points (not the airplane icon Google recently switched to).
    • The erdCalcofi datasets were re-arranged and are now served from local files (faster).
    • GenerateDatasetsXml fromThreddsCatalog now creates a results file:
      [tomcat]/webapps/erddap/WEB-INF/temp/EDDGridFromThreddsCatalog.xml . Thanks to Kevin O'Brien.
    • GenerateDatasetsXml fromThreddsCatalog now tries to remove unnecessary port numbers from the source URLs (e.g., :8080 and :8081 can sometimes be removed). Thanks to NOAA central's security team.
    • For .subset web pages, the Map of Distinct Data now has a variable lat lon range.
    • Several lists in ERDDAP (e.g., the table which shows all of the datasets) were sorted so that A..Z sorted before a..z. Now they sort in a case-insensitive way.
    • Small changes to the .subset web pages, including: units are now indicated.
    • GenerateDatasetsXml and DasDds no longer throw an exception if unable to put the results on the system clipboard or displayInBrowser. Thanks to Eric Bridger and Greg Williams.
    • Bug fix: When datasets are loaded, ERDDAP now removes or adjusts the geospatial global attributes. Thanks to Charles Carleton.
    • Bug fix: String2.getClassPath() now properly percent-decodes the classPath (notably, on Windows, spaces in the file name appeared as %20). This affected ERDDAP EDStatic calling SSR.getContextDirectory() and finding content/erddap. Thanks to Abe Coughlin.
    • Bug fix: in EDDTableFromFiles related to getDataForDapQuery handling of distinct() requests. Thanks to Eric Bridger.
    • Bug fix: tabledap requests didn't properly handle altitude constraints when the dataset's altitudeMetersPerSourceUnit was -1. Thanks to Eric Bridger.
    • Bug fix: EDDTableFrom...Files datasets now correctly handle requests which include =NaN and !=NaN.

     

Changes in version 1.28 (released 2010-08-27)

  • New Features: none.
  • Things ERDDAP Administrators Need to Know and Do: none.
  • Bug Fix: Fix a programming mistake (only in ver 1.26) that made ERDDAP very slow.
     

Changes in version 1.26 (released 2010-08-25)

  • New Features: none.
  • Things ERDDAP Administrators Need to Know and Do:
    • From your [tomcat]/content/erddap/setup.xml,
      • In <legal>, on a new line below [standardDataLicenses], insert [standardContact]. [standardContact] refers to the <adminEmail> specified higher up in setup.xml.
      • Remove <tableCommonBGColor> and <tableHighlightBGColor>.
      • Recommended: Change <endBodyHtml> to
        <endBodyHtml><![CDATA[
        <br>&nbsp;
        <hr>
        ERDDAP, Version &erddapVersion;
        <br><a href="&erddapUrl;/legal.html">Disclaimers</a> |
        <a href="&erddapUrl;/legal.html#privacyPolicy">Privacy Policy</a> |
        <a href="&erddapUrl;/legal.html#contact">Contact</a>
        </body>
        ]]></endBodyHtml>
    • Required: To your [tomcat]/content/erddap/images/erddap.css and erddapAlt.css, add at the bottom:
      /* This is used on the /info/[datasetID]/index.html pages to highlight a row or cell. */
      tr.highlightBGColor {background-color:#cceecc; }
      td.highlightBGColor {background-color:#cceecc; }
  • Bug Fixes and Small Changes:
    • Bug fix: in some situations, forms didn't work in some versions of Internet Explorer. Thanks very much to Greg Williams.
    • Bug fix: The Make A Graph buttons didn't work if the dataset was from a remote ERDDAP.
    • Bug fix: WMS sometimes didn't work if the dataset was from a remote ERDDAP.
    • Many small changes and bug fixes.

     

Changes in version 1.24 (released 2010-08-06)

  • New Features:
    • New Subset web pages use faceted search to select subsets of tabular datasets. Thanks to POST.
    • New Advanced Search combines all of the other search options and adds longitude, latitude, and time bounding boxes. Thanks to Ellyn Montgomery. (Sorry for the delay.)
    • New Convert Time web page and service let you convert numeric times to/from ISO string times.
    • New Convert Units web page and service let you convert UDUNITS to/from UCUM units. Thanks to NOAA IOOS SOS.
    • If a tabledap request includes &units("UCUM"), the units names will be converted from original names (usually UDUNITS) to UCUM units names. This only affects units *names*, not data values. Thanks to NOAA IOOS SOS.
    • Improvements to Make A Graph web pages and graphs and maps:
      • If the graph is a map, there are new Make A Graph buttons to zoom in/out and a new option to click to change the map's center point. Thanks to POST.
      • Filter settings added near the bottom. Thanks to Greg Williams.
      • The built in coastline data files were updated to GSHHS v2.0. Thanks to POST.
      • Maps now include lakes and rivers. Thanks to POST. (Sorry, the Sacramento River Delta is missing because neither the coastline data nor the lake/river dataset deals with it.)
      • The built in pscoast-derived nation/state files were updated. Thanks to POST.
      • Topography.cpt was modified slightly. (Sorry if this adversely affects you.) Thanks to POST.
      • In griddap's Make A Graph, if a user changes a variable, the form is automatically resubmitted so that the axisVariables' showStartAndStop always reflects the graph variables. Thanks to Joaquin Trinanes.
      • For png and pdf image URLs:
        • New &.land=value, where value can be "under" (show topography) or "over" (just show bathymetry). If not specified, the default is set by drawLandMask in datasets.xml or setup.xml. Thanks to POST.
        • New: lines in the legend that are too long are automatically broken into multiple lines. Thanks to POST.
      • For png image URLs:
        • New &.legend=value, where value can be "Bottom" (default), "Off" or "Only". This lets you include the legend, exclude the legend, or get only the legend. Thanks to Cara Wilson.
        • New &.trim=nPixels leaves a border of nPixels (e.g., 10) at the bottom of the image. It is applied after .legend=Off. Thanks to Cara Wilson.
        • New &.size=width|height lets you specify the width and height for the image, in pixels.
    • New output file formats:
      • .csvp and .tsvp - like .csv and .tsv, but with "(units)" appended to column names on first line.
      • .odvTxt - makes a .txt file that simplifies getting data into Ocean Data View (ODV).
      • .esriCsv - makes a .csv file suitable for import in ESRI's ArcGIS. (tabular datasets only)
      Thanks to Jan Mason, Jeff de La Beaujardiere, and NOAA IOOS SOS project.
    • GUI improvements to the Categorize web pages. Also, the categorize values (other than institution) are now all lower case. Non-lower case requests are accepted (redirected) for backwards compatibility. Thanks to Roy Mendelssohn.
    • Error messages are now even shorter and more oriented to users. Thanks to Greg Williams.
    • An internal change which greatly reduces ERDDAP's base memory usage.
    • Many new features which are only relevant to the POST project.
  • Things ERDDAP Administrators Need to Know and Do: There are lots of changes. Sorry. But each one brings some nice benefit.
    • Big changes to GenerateDatasetXml - it now often asks more questions (see the relevant datasetTypes information) and now always generates essentially ready-to-use content for datasets.xml. You are still responsible for the setup, so you should still review the datasets.xml content before using it. A human putting effort into the project will always do better than a computer program. Thanks to the UAF project.
    • REQUIRED: In setup.xml, you must revise the WMS section. It should now include these tags (but feel free to change the values):
      <!-- These default accessConstraints, fees, and keywords are used 
      by the SOS, WCS, and WMS services.
      They can be overridden by "accessConstraints", "fees", "keywords" 
      attributes in a dataset's global metadata.
      If a dataset that has an "accessibleTo" tag doesn't override 
      "accessConstraints", then the default for "accessConstraints" is the
      "accessRequiresAuthorization" value.  
      -->
      <accessConstraints>NONE</accessConstraints>
      <accessRequiresAuthorization>only accessible to authorized
      users</accessRequiresAuthorization>
      <fees>NONE</fees>
      <keywords>Earth science, oceans</keywords> 
      
      <!-- This appears on the erddap/legal.html web page after the 
      General Disclaimer. 
      You can replace any of the [standardParts] with your own HTML. -->
      <legal><![CDATA[
      [standardDisclaimerOfEndorsement]
      [standardDisclaimerOfExternalLinks]
      [standardPrivacyPolicy]
      [standardDataLicenses]
      ]]></legal>
      
      <!-- Specify the default units standard (e.g., "UDUNITS" 
      (the default) or "UCUM") that you (the ERDDAP admin) are using to 
      specify units.  The value is case-sensitive.
      This is used by ERDDAP's SOS server to determine if the units need to
      be converted to UCUM units for WMS and SOS GetCapabilities responses. 
      -->
      <units_standard>UDUNITS</units_standard>
      
      <!-- For the wms examples, pick one of your grid datasets that has
      longitude and latitude axes.
      The sample variable must be a variable in the sample grid dataset.
      The bounding box values are minx,miny,maxx,maxy.
      -->
      <wmsSampleDatasetID>erdBAssta5day</wmsSampleDatasetID>
      <wmsSampleVariable>sst</wmsSampleVariable>
      <!-- The bounding box values are 
         minLongitude,minLatitude,maxLongitude,maxLatitude.
         Longitude values within -180 to 180, or 0 to 360, are now okay. -->
      <wmsSampleBBox>0,-75,360,75</wmsSampleBBox>
      
    • REQUIRED: In setup.xml, copy and paste this new suggested <startHeadHtml> to replace your old version. But feel free to make changes for your preferences.
      <!-- startHeadHtml has the start of the HTML document and the 
      'head' tags (starting at "<!DOCTYPE>", but not including 
      "</head>") for all HTML web pages. 
      This may include &erddapUrl;, which is expanded to be 
        [baseUrl]/erddap (or [baseUttpsUrl]/erddap if the user is logged in).
      If your ERDDAP allows users to log in, all referenced image files, 
        css files, etc. must be in [tomcat]/content/erddap/images or a 
        subdirectory and must be referenced here with 
        &erddapUrl;/images/[fileName].
      
      favicon.ico is the image that browsers associate with your website.
      More more information, see http://en.wikipedia.org/wiki/Favicon .
      You can use your own favicon.ico file by putting it in 
        [tomcat]/content/erddap/images. 
      
      *** Optional: you can change the appearance of all of your 
      ERDDAP's HTML pages by changing the CSS <style> settings below.
      
      For an example of a very different style, change the import reference
      to <tomcat>/content/erddap/images/erddapAlt.css
      
      *** If your CSS style includes links to files (e.g., images), that 
      style information must be inline in the style tag below, after the
      'import' line, not in the .css file.  
      Put all of the (e.g., image) files in the 
      [tomcat]/content/erddap/images directory (or a subdirectory) and 
      reference them below starting with &erddapUrl;.
      Why? On ERDDAP https: web pages, *all* links should use "https:" 
      (not "http:"); otherwise, most browsers consider the web page not 
      fully secure.  Because ERDDAP would use the same .css file for 
      http: and https: web pages, the links within the .css file wouldn't 
      switch between http: and https:.  There doesn't seem to be a way 
      around this other than using inline style information.
      -->
      <startHeadHtml><![CDATA[ 
      <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
        "http://www.w3.org/TR/html4/loose.dtd">
      <html>
      <head>
      <title>ERDDAP</title>
      <link rel="shortcut icon" href="&erddapUrl;/images/favicon.ico">
      <style type="text/css">
      <!--
        @import "&erddapUrl;/images/erddap.css";
      -->
      </style>
      ]]></startHeadHtml>
      
      <!-- The tableCommonBGColor MUST be the same color as the 
         table.commonBGColor in erddap.css above. Suggested is #f1ecd8. 
         But if you use erddapAlt.css, change this to #e7dec5. -->
      <tableCommonBGColor>#f1ecd8</tableCommonBGColor>
      
      <!-- This is used, e.g., for the type=variable rows on the metadata
        info tables. -->
      <tableHighlightBGColor>#cceecc</tableHighlightBGColor>
      
      Thanks to POST, Hans Vedo, and Rick Blair.
    • REQUIRED: In setup.xml, in <startBodyHtml>, change the <body> tag to be just <body>, since the style is now set by erddap.css.
    • REQUIRED: In setup.xml, change to this <endBodyHtml> (but change the email address to your email address and feel free to make other changes):
      <!-- The end of the body of the HTML code for all HTML web pages
        (with "</body>" at the end). 
      This may include &erddapUrl;, which is expanded to be 
        [baseUrl]/erddap (or [baseUttpsUrl]/erddap if the user is logged in).
      If your ERDDAP allows users to log in, all referenced image files, etc. 
        must be in [tomcat]/content/erddap/images or a subdirectory
        and must be referenced here with &erddapUrl;/images/[fileName].
      
      You can change this, but please keep "ERDDAP, Version &erddapVersion;"
      and these references to the Disclaimers and Privacy Policy. -->
      <endBodyHtml><![CDATA[ 
      <br>&nbsp;
      <hr>
      ERDDAP, Version &erddapVersion;
      <br><font class="subduedColor">Questions, comments, 
        suggestions?  Please send an email to 
        <tt>bob dot simons at noaa dot gov</tt>
      <br>and include the ERDDAP URL directly related to your question
        or comment.
      <br>
        <a href="&erddapUrl;/legal.html">Disclaimers</a> | 
        <a href="&erddapUrl;/legal.html#privacyPolicy">Privacy 
          Policy</a>
      </font>
      </body>
      ]]></endBodyHtml>
      
    • HIGHLY RECOMMENDED: In setup.xml, the recommended <theShortDescriptionHtml> is now
      <theShortDescriptionHtml><![CDATA[ 
      <h1>ERDDAP</h1>
      This web site (the Environmental Research Division's Data Access 
      Program) aggregates scientific data from diverse local and remote 
      sources and offers you a simple, consistent way to download subsets 
      of the data in common file formats and make graphs and maps.
      This particular ERDDAP installation has oceanographic data
      (for example, data from satellites and buoys).
      
      [standardShortDescriptionHtml]
      ]]></theShortDescriptionHtml>
      
      Feel free to change this, particularly the last sentence in the first paragraph.
    • In setup.xml, emailEverythingTo and emailDailyReportTo can now be comma-separated lists of email addresses. The first emailEverythingTo is special, e.g., subscriptions to EDDXxxxFromErddap datasets use that email address. Thanks to John Maurer.
    • Email errors are now logged to the [bigParentDirectory]/logs/emailLogYYYY-MM-DD.txt file.
    • In setup.xml, there is a new, optional parameter to set email account properties (usually right after <emailPassword>):
        <emailProperties>propertyName1|propertyValue1|propertyName2| propertyValue2|...</emailProperties>
      For example, gmail accounts need
        <emailProperties>mail.smtp.starttls.enable|true</emailProperties>
      The default is nothing. Thanks to Rich Signell.
    • REQUIRED: If you use EDDTableCopy or EDDGridCopy, you must DELETE all [bigParentDirectory]/copy/ directories and files that contain "xh" in the directory or file names after stopping the old ERDDAP and before starting the new ERDDAP so those files will be re-copied. I'm very sorry, but it was important to make the change and hopefully it affects few admins and few files.
      In Linux, you can find these files with,
        cd [bigParentDirectory]/copy
        find . *xh*

      In Windows, you can find these files with,
        Start | Search
         What do you want to search for: Documents
          All or part of the file name: xh
          Look in: Browse -> [bigParentDirectory]/copy
          Click on 'Search'
           ^A to select them all
           Del to delete them all
    • REQUIRED: In datasets.xml, for EDDTableFromDatabase datasets, for date and timestamp variables, change the dataType to double and the units to seconds since 1970-01-01T00:00:00Z. We REQUIRE that you store timestamp data in the database *with* a timezone. Without timezone information, the queries that ERDDAP sends to the database and the results that ERDDAP gets from the database via JDBC are ambiguous and are likely to be wrong. We tried, but found no reliable way to deal with "timestamp without timezone" data. We think this is good practice anyway. After all, "timestamp without timezone" data has an implied timezone. While it is great that the time zone is obvious to the database admin, it makes sense to specify it explicitly so that other software can properly interact with your database. Thanks/sorry Michael Urzen.
    • HIGHLY RECOMMENDED: In datasets.xml, to enable .subset web pages for faceted search of your tabular datasets, you need to add <subsetVariables> to the dataset's global attributes.
    • RECOMMENDED: In datasets.xml, if you have the dataset with datasetID="pmelGtsppp", please change it to be
        <dataset type="EDDTableFromDapSequence" datasetID="pmelGtsppp" active="false">
      Whether or not you had that dataset, feel free to add this new GTSPP dataset:
        <dataset type="EDDTableFromErddap" datasetID="erdGtsppBest">
          <sourceUrl>http://coastwatch.pfeg.noaa.gov/erddap/tabledap/erdGtsppBest</sourceUrl>
        </dataset>
    • RECOMMENDED: In datasets.xml, there are new valid options for the <cdm_data_type> global attribute, so you should review/change the value for your datasets.
    • In datasets.xml, the new <sourceNeedsExpandedFP_EQ> is helpful if the source server doesn't consistently handle &variable=value tests correctly (because of the general difficulty of testing the equality of floating point numbers). sourceNeedsExpandedFP_EQ is set to true by default (the safest setting), so you don't need to make any changes.
    • New EDDTableFromAsciiFiles. Thanks to Jerry Yun Pan.
    • New EDDTableFromThreddsFiles. Thanks to Roy Mendelssohn.
    • Changes to EDDTableFromNcFiles lets it be used with a wider range of files.
    • EDDTableFromBMDE has been disabled. There are no longer any active, appropriate, data sources.
    • In GenerateDatasetXml, the new EDDGridFromThreddsCatalog harvests an entire THREDDS catalog (or a subset) and generates datasets.xml content. Thanks to the UAF project.
    • GenerateDatasetsXml and DasDds now also put their results in [bigParentDirectory]/logs/log.txt. Thanks to Rich Signell and Charles Carleton.
    • Many improvements to the login system. Thanks to POST.
  • Things ERDDAP Programmers Need to Know and Do:
    • There have been changes in the /WEB-INF/lib/ directory. Please change your javac and java classpath settings accordingly.
    • There is a new [yourUrl]/erddap/version service to determine the version of an ERDDAP. The response is text, e.g., ERDDAP_version=1.24 If you get an HTTP 404 Not-Found error message, treat the ERDDAP as version 1.22 or lower. Thanks to POST.
  • Small Changes and Bug Fixes:
    • EDDTableFromSos changes:
      • Dropped support for reading IOOS SOS XML responses.
      • Added support for reading IOOS SOS text/csv. (So NOS SOS servers currently aren't supported.)
      • Made lots of changes related to IOOS SOS server details.
      • Added support for BBOX queries for IOOS SOS and OOSTethys SOS servers.
      These changes result in a big speed up for relevant data requests. Thanks to IOOS SOS.
    • Text in .mat tabular data files is now saved correctly. Thanks to Roy Mendelssohn.
    • WMS
      • OpenLayers is now bundled with ERDDAP for use on the WMS web pages. This fixes the problem caused when OpenLayers changed a few months ago and prevents future problems.
      • In the WMS GetCapabilities response, the <OnlineResource> value is now the URL of the WMS service. Thanks to Charlton Galvarino.
      • A legend is displayed on the WMS web page to show the colorbar. Thanks to Emilio Mayorga.
    • EDDGridAggregateExistingDimension constructor had problems if an axis' sourceValues weren't equal to their destinationValues, e.g., if source time was something other than "seconds since 1970-01-01". Thanks to Todd Spindler.
    • In TableWriterGeoJson, the excess ',' after bbox[...] as been removed. Thanks to Greg Williams.
    • Many small changes and bug fixes.

     

Changes in version 1.22 (released 2009-07-05)

  • The SlideSorter bug introduced in 1.20 is fixed.
  • The OBIS bug introduced in 1.20 is fixed.
  • The references to Jason datasets on the images/gadgets/GoogleGadgets page were removed.
     

Changes in version 1.20 (released 2009-07-02)

  • ERDDAP administrators, please add this to your setup.xml file:
    <!-- If you want to restrict access to some datasets, you need to 
    specify the method used for logging on (authentication). See the info 
    at http://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security
    Currently, the options are: "" (logins not supported, the default),
    "custom", "openid". Note that openid login doesn't work when testing 
    with localhost (https://127.0.0.1:8443).
    -->
    <authentication></authentication>
    
    <!-- This specifies how you have stored passwords in the roles tags 
    in datasets.xml. If you aren't storing any passwords this is irrelevant.
    The options (in order of increasing security) are: "plaintext", "MD5", 
    or "UEPMD5" (MD5(UserName:ERDDAP:Password), the default).
    You should only use "plaintext" or "MD5" if you need to match values 
    stored that way in an external password database.  See the info at
    http://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security
    -->
    <passwordEncoding>UEPMD5</passwordEncoding>
    
    <!-- This determines whether datasets that the user doesn't currently
    have access to (because he isn't logged in or because his roles don't
    allow access) should be shown on lists of data sets 
    (e.g., from full text search, categorize, view all datasets, ...).
    The options are: "true", or "false" (the default).
    If false, no information about the dataset (even its existence) is 
      shown to users who don't have access to it.
    If true, some information about the dataset (title, summary, etc) is
      shown to users who don't have access to it.  
      If the user clicks on a link to a dataset he doesn't have access to,
      he will get an error message and be prompted to log in.
    -->
    <listPrivateDatasets>false</listPrivateDatasets>
    
    <!-- If the number of requests between two runs of LoadDatasets 
    exceeds unusualActivity, an email is sent to emailEverythingTo.
    The default is 10000.
    -->
    <unusualActivity>10000</unusualActivity>
    
  • New dataset types EDDGridCopy and EDDTableCopy make and maintain a local copy of another EDDGrid or EDDTable dataset's data and serve data from the local copy. These are very easy to use and very effective solutions to some of the biggest problems with serving data from remote data sources:
    • Accessing data from a remote data source can be slow (for a variety of reasons).
    • The remote dataset is sometimes unavailable (again, for a variety of reasons).
    • Relying on one source for the data doesn't scale well (e.g., when many users and many ERDDAPs utilize it).
    Plus, the local copy is a backup of the original, which is useful in case something happens to the original.

    There is nothing new about making a local copy of a dataset. What is new here is that these classes make it *easy* to create and *maintain* a local copy of data from a *variety* of types of remote data sources and *add metadata* while copying the data.

    These dataset types are part of a complete set of features that simplify the creation of grids/clusters/federations of ERDDAPs to handle very heavy loads (e.g., in a data center).

  • New dataset type EDDTableFromDatabase gets data from a local or remote database table.
  • ERDDAP now has a security system that supports authentication (letting users log in) and authorization (granting them access to certain private datasets).
  • There are two, new, command-line tools to help ERDDAP administrators generate the XML for a new dataset in datasets.xml:
    • GenerateDatasetsXml can generate a rough draft of the dataset XML for almost any type of datasets.
    • DasDds helps you repeatedly test and refine the XML for a dataset.
    ERDDAP's GenerateDatasetsXml web pages have been removed. For security reasons, they only supported a few dataset types. The new command line tools are a better solution.
  • The new status page lets anyone (but especially administrators) view the status of an ERDDAP from any browser by going to [baseUrl]/erddap/status.html .
  • Tabledap now supports server-side functions:
    • &distinct() removes duplicate rows from the response table,
    • &orderBy(...) lets you specify how the response table should be sorted,
    • &orderByMax(...) lets you specify how the response table should be sorted and removes all rows except for the rows with the maximum values in the last specified column. This can be used, for example, to get the last available data for each station.
  • Tabular datasets can now include additional dateTime variables which aren't named "time". These variables are recognized by their "units" metadata, which must contain " since " (for numeric dateTimes) or "yy" or "YY" (for formatted String dateTimes). But please still use the destinationName "time" for the main dateTime variable.
  • ERDDAP now generates a sitemap.xml file, which tells search engines that your ERDDAP only needs to be crawled every month. ERDDAP administrators, please follow these instructions to notify the search engines about the new sitemap.xml file.
  • ERDDAP's error messages are now much shorter and geared to clients (not programmers). Thanks to Greg Williams.
  • <requestBlacklist> now also supports IP addresses where the last number has been replaced by *.
  • Requests for .json and .geoJson files may now include an optional jsonp request by adding "&.jsonp=functionName" to the end of the query. Basically, this just tells ERDDAP to add "functionName(" to the beginning of the response and ")" to the end of the response. If originally there was no query, leave off the "&" in your query. Thanks to Greg Williams.
  • Lots of new statistics were added to the Daily Report.
  • On web pages with lists of datasets, institution and id are now at the far right. This moves subscription and other more useful columns into view on narrow computer screens.
  • On all web pages, the page's title (based on the <title> in the <startHeadHtml> that you define in setup.xml) is modified to include a better description of the web page (for example, by including the current dataset's title and institution).
  • Xmx information is now included with the memory information printed in log.txt, the Daily Report, and on status.html. Thanks to Ellyn Montgomery.
  • Erddap has additional, general-purpose protection against all errors (e.g., OutOfMemoryError). Thanks to Charles Carleton.
  • Improvements to error handling if the response has already been committed.
  • Change: EDDTableFromFiles and EDDGridFromFiles now just allow <metadataFrom> first or last. penultimate is no longer supported. And first and last are now based on the files' lastModifiedTime.
  • Bug fix: in EDDTableFromSOS, invalid info for one station threw an exception and caused the whole dataset to be rejected. Now, those stations are just ignored (and the error message is logged to log.txt). Thanks to Rick Blair.
     

Changes in version 1.18 (released 2009-04-08)

  • Bug fix: Starting in 1.14, the EDDTable Data Access Form and Make A Graph web page didn't properly deal with quoted constraints.
  • Bug fix: Starting in 1.14, EDDTableFromDapSequence didn't handle time constraints correctly if the source time units weren't "seconds since 1970-01-01T00:00:00".
     

Changes in version 1.16 (released 2009-03-26)

  • ERDDAP administrators:
    • This is an important release because it fixes a bug that left an ERDDAP thread running if you used Tomcat Manager to Stop/Start or Reload ERDDAP. So when you install 1.16, don't just use Tomcat manager to undeploy the old ERDDAP and deploy the new ERDDAP. Instead: undeploy the old ERDDAP, restart Tomcat (or the server), then deploy the new ERDDAP. It's always a good idea to do that when installing a new version.
    • Please add <requestBlacklist></requestBlacklist> to your datasets.xml. This can be used to specify a list of client IP addresses to be blocked (e.g., to fend off a Denial of Service attack or an overly zealous web robot).
  • There is now a [bigParentDirectory]/logs directory to hold the ERDDAP log files. When you start ERDDAP, it makes an archive copy of the log.txt and log.txt.previous files with a time stamp. If there was trouble before the restart, it may be useful to analyze these files.
  • ERD's ERDDAP now has the subscription system turned on.
  • ERDDAP once again allows (but still doesn't recommend) the "%26" encoding of "&" in request URLs (see the related v1.14 change).
  • Several new additions to the Tally section of the Daily Report.
  • Small bug fixes in generateDatasetsXml.
  • A few small bug fixes.
     

Changes in version 1.14 (released 2009-03-17)

  • Changes for users:
    • In grid data requests, ERDDAP now supports: last-n where n is an integer number of indices and (last-d) where d is a numeric value (for time, it is in seconds).
    • In tabular data requests, String constraints now require double quotes around the value, for example, &id="NDBC40121" This is required by the DAP protocol.
    • In tabular data requests, ERDDAP now requires that all constraints be properly percent encoded. Browsers do this automatically, so this mostly affects computer programs/scripts that are accessing ERDDAP.
    • Previously, the embed a graph web page and the ERDDAP Google Gadget web page said to replace the "&" in the image's URL with "%26". From now on, you should replace the "&" in the image's URL with "&amp;". So you need to replace any "%26" in existing web pages and Google Gadgets with "&amp;". (Sorry)
  • ERDDAP administrators, please:
    • Add the following to your setup.xml file (and change the flagKeyKey value):
      <!-- ERDDAP has a service that lets remote users set a flag
      to notify ERDDAP to try to reload a dataset.
      These requests use a key which is generated based
      on baseUrl/warName, a datasetID, and flagKeyKey.
      *** Change this once, to any text (a favorite quote? random text? 
      It doesn't matter). Normally, you won't ever change this again.
      But if you you think someone is abusing the flag system,
      change this text again, restart ERDDAP, and send
      all of the users of the flag system the relevant new flagKeys
      (see the list in the Daily Report). -->
      <flagKeyKey>A stitch in time saves nine. CHANGE THIS!!!</flagKeyKey>
      
      <!-- ERDDAP has an email/URL subscription system which sends a user
      an email or pings a url whenever a dataset of interest changes.
      (This is different from the RSS system, which is always active.)
      The system relies on the server being able to send out 
      emails to people to validate their subscription requests.
      The emails appear to come from the emailFromAddress below.
      So if your server can't send out emails, don't make this system active.
      You may choose (for whatever reason) to make this system active or not, 
      so valid values below are "true" (the default) and "false".
      Note that if you change this and restart ERDDAP, the list of 
      subscriptions (in [bigParentDirectory]/subscriptionsV1.txt) isn't
      affected. See also the subscriptionEmailBlacklist in datasets.xml.
      -->
      <subscriptionSystemActive>true</subscriptionSystemActive>  
    • On the line after <emailUserName> in in setup.xml, add
      <emailPassword>myPassword</emailPassword> <!-- optional; if absent, emails can't be sent to non-local addresses >
      and enter your real password.
    • You can change <wmsSampleBBox> in setup.xml to include longitude values up to 360, e.g.,
      <!-- The bounding box values are 
         minLongitude,minLatitude,maxLongitude,maxLatitude.
         Longitude values within -180 to 180, or 0 to 360, are now okay. -->
      <wmsSampleBBox>0,-75,360,75</wmsSampleBBox>  
    • In your datasets.xml file, rename dataset type EDDTableFromNc4DFiles to EDDTableFromNcFiles (which now supports files with any number of dimensions). If you had an EDDTableFromNc4DFiles dataset:
      1. You MUST change to type="EDDTableFromNcFiles" in your datasets.XML file.
      2. You MUST add a <nDimensions>4</nDimensions> tag to the dataset's XML.
      3. You may add the new <sortFilesBySourceNames> tag to specify the internal order for the files, which determines the overall order of the data returned.
      For details, see EDDTableFromFiles.
    • In the past, for EDDTableFromDapSequence, for OPeNDAP DRDS servers, in datasets.xml, we used <sourceCanConstrainStringsRegex>~=</sourceCanConstrainStringRegex>. But we now see that the DRDS regex support is more limited than ERDDAP's, so we recommend <sourceCanConstrainStringsRegex></sourceCanConstrainStringRegex> so that regex constraints are not passed to the source, but are instead handled by ERDDAP.
    • Revamped handling of sourceCanConstrain... in datasets.xml by EDDTableFromDapSequence and (internally) all EDDTable dataset types. The new system is simpler and better reflects the variability of different data sources. You may need to modify the XML for your datasets in datasets.xml.
  • There are several new features which are useful by themselves, but when combined, also facilitate the creation of grids/clusters/federations of ERDDAPs.
    • New dataset types:
    • RunLoadDatasets and LoadDatasets were revamped so that ERDDAP is very responsive to reloading datasets based on files in the flag directory (often <5 seconds if main loadDatasets is currently done).
    • New service to allow a URL to create a flag file for a given dataset, e.g.,
      http://coastwatch.pfeg.noaa.gov/erddap/setDatasetFlag.txt?datasetID=rPmelTao&flagKey=123456789
      creates a flag file in the flag directory for rPmelTao (although the flagKey here is wrong).
    • New subscription service so that any client can specify an action which will be done when a specific dataset is created (when ERDDAP is restarted) and whenever the dataset changes in any way. This system can be disabled via <subscriptionSystemActive> in setup.xml. The ERDDAP Daily Report now lists all of the subscriptions and includes the URL needed to cancel each one, in case you feel the system is being abused. In datasets.xml, there is a new, optional <subscriptionsEmailBlacklist> tag so that administrators can specify a comma-separated list of email address which are immediately blacklisted from the subscription system.
    • New <onChange> attribute in datasets.xml lets the ERDDAP administrator specify an action which will be done when a specific dataset is created (when ERDDAP is restarted) and whenever the dataset changes in any way.
    • Improvements to full text search: storing the search string for each dataset now uses 1/2 the memory. The search algorithm (Boyer-Moore-like) is now 3X faster.
    • Emails from ERDDAP now always prepend the subject and content with [erddapUrl], so that it will be clear which ERDDAP this came from (in case you administer multiple ERDDAPs).
    • More extensive statistics gathering for the Daily Report email.
    • New log file [bigParentDirectory]/emailLogYEAR-MM-DD.txt logs all emails sent by ERDDAP each day. This is especially useful if your server can't actually send emails -- you can at least read them in the log.
    • ERDDAP now makes a [bigParentDirectory]/cache/(datasetID) directory for each dataset since there may be lots of files cached.
  • New RSS 2.01 feed for each dataset (look for the orange RSS icons on lists of datasets, Data Access Forms, and Make A Graph web pages).
  • EDDGrid .kml responses now use tiled images ("superoverlays" - dynamically generated quadtree images). The initial image loads into GoogleEarth much faster than before. The resolution of the map increases as you zoom in, up to the full resolution of the dataset. Recommend: users should request .kml for one time point, but the dataset's entire longitude,latitude range. Unfortunately, support for time ranges was removed (I hope it will come back).
  • ERDDAP now adds Expires and Cache-Control max-age headers to all files requested from the /images directory. This greatly reduces the number of static file requests sent to ERDDAP and thus greatly speeds up most ERDDAP page loads. Also, many JavaScript file references moved to the bottom of their HTML pages, which also speeds up many ERDDAP page loads. Thanks to the book "High Performance Web Sites" by Steve Souders and the ySlow addition to the FireBug plugin in FireFox.
  • ERDDAP switched from netcdf-java 2.2.22 to netcdf-java 4.0. Among other things, this allows EDDGridFromNcFiles to read HDF .hdf, as well as GRIB .grb and NetCDF .nc files.
  • EDDGridFromDap and EDDGridFromNcFiles now also support DArray (as well as DGrid) dataVariables. If a dimension doesn't have a corresponding coordinate variables, ERDDAP creates an axis variable with the index values (e.g., 0, 1, 2, ..., 311, 312). So all other aspects of EDDGrid remain the same:
    * It still serves all datasets as Grids, with an axis variable for each dimension.
    * Queries can still request values from the axis variables.
    Thanks to Charles Carleton, Thomas Im, Dorian Raymer, and others.
  • The WMS OpenLayers pages now have a default longitude,latitude range that is a little larger than the dataset's range (not the exact range, so the context of small datasets is more obvious). The default range may now also be 0 to 360, which allows the full range of many datasets to be shown now. Thanks to Todd Spindler.
  • New sliders on some Data Access Forms and Make A Graph web pages. They simplify (crude) specification of the desired data and offer good visual feedback.
  • A new option for the <dataset> tags in datasets.xml: active="false".
  • References to ERD's ERDDAP changed from coastwatch.pfel (still works via proxy) to coastwatch.pfeg (preferred).
  • New support for data_min and data_max variable metadata attributes.
  • A partial solution to the WaitThenTryAgain / Partial Results Exception: Now, some requests that previously failed when a data source change was detected will succeed because ERDDAP will reload the dataset and re-request the data automatically, all in the context of the original request.
  • Bug fix: generateDatasetsXml was disabled in ERDDAP version 1.12. Thanks to Ellyn Montgomery for pointing this out.
  • Small changes to error handling.
  • Many improvements to avoid/deal with possible race conditions (i.e., possible problems arising from the multi-threaded nature of ERDDAP) which caused small, infrequent problems.
  • Now, if an error message is written on an image, the image will only stay in the cache for ~5-10 minutes (not 60). Thanks to Cara Wilson.
  • The standard message when there is no data is now "Your query produced no matching results.", which is shorter, more accurate, and matches OPeNDAP servers.
  • EDDGrid no longer allows tied axis values.
  • Small changes to .ver and .help requests.
  • Many small changes and bug fixes.
     

Changes in version 1.12 (released 2008-10-31)

  • EDDTableFromSOS once again works with NDBC SOS and works with the new NOS SOS.
  • EDDTableFromBMDE now requires ERDDAP admin to specify dataVariables.
  • EDDGrid no longer requires that lat and lon be evenly spaced for .transparentPng or .kml. Thanks to Todd Spindler.
  • A few small changes.
     

Changes in version 1.10 (released 2008-10-14)

  • New "colorBar" metadata for data variables in datasets.xml defines the default color bar settings for graphs and maps. See more information. This is important because it greatly improves the appearance of the default graphs and maps produced by Make A Graph and because the default graphs and maps now have a consistent color bar even when the client changes the requested time or geographic range. Also, this was necessary for WMS.
  • ERDDAP now serves most grid data via a WMS service. This is important because it shows that, in addition to getting data from many types of data servers, ERDDAP can distribute data via different protocols (DAP, WMS, ... more in future). See the client documentation. Or the documentation for administrators. Or try it out.
  • New support for longitude values >180 in .kml files.
  • New cdm_data_type: Other .
  • ERDDAP now supports "boolean" source dataType. See more information This will become useful for the future EDDTableFromDatabase.
  • New EDDTableFromBMDE supports DiGIR/BMDE data sources.
  • EDVGridAxis now allows descending sorted values. The pmelOscar datasets needed this.
  • ERDDAP now returns HTTP errors (e.g., "404 for resource/page not found") in more situations, instead of HTML pages with error messages.
  • Lots of changes/additions to the ERDDAP documentation.
  • Lots of small changes.
  • Some bug fixes.
  • Things ERDDAP administrators should do to upgrade to this version:
    • In datasets.xml, for any EDDTableFromSOS datasets, change "observedProperty" metadata to "sourceObservedProperty".
    • The rules for an axisVariable or dataVariable's destinationName are now stricter. You need to check that your variable names are valid. Either check them by hand, or run ERDDAP and look at the error messages in the report that is emailed to the administrator.
    • In datasets.xml, if you want a grid data variable to be accessible via WMS, you need to add colorBar metadata. At least, for example,
        <att name="colorBarMinimum" type="double">0</att>
        <att name="colorBarMaximum" type="double">32</att>

      See more information.
    • Add the following to your setup.xml (but customize with your information):
      
      <!-- drawLand specifies the default Make A Graph setting for 
      whether the landmask should be drawn "over" (the default) or "under" 
      surface data on maps. "over" is recommended for primarily 
      oceanographic data (so that grid data over land is obscured by the 
      landmask). "under" is recommended for all other data.
      -->
      <drawLand>over</drawLand>  
      
      <!-- Information about the ERDDAP administrator is used for the 
      SOS and WMS servers. You MUST CHANGE these to describe your 
      installation. 
      -->
      <adminInstitution>NOAA Environmental Research 
      Division</adminInstitution>
      <adminIndividualName>Your Name</adminIndividualName>
      <adminPosition>Webmaster</adminPosition>
      <adminPhone>your-phone-number</adminPhone>
      <adminAddress>1352 Lighthouse Ave.</adminAddress>
      <adminCity>Pacific Grove</adminCity>
      <adminStateOrProvince>CA</adminStateOrProvince>
      <adminPostalCode>93950</adminPostalCode>
      <adminCountry>USA</adminCountry>
      <adminEmail>yourName@yourSite</adminEmail>
      
      <!-- Information about the ERDDAP administrator is used for ERDDAP's
      SOS server. You MUST CHANGE these to describe your installation. 
      -->
      <sosTitle>NOAA Environmental Research Division SOS</sosTitle>
      <sosAbstract>NOAA Environmental Research Division's ERDDAP makes 
        data from multiple sources available via the SOS 
        protocol.</sosAbstract>
      <sosKeywords>Weather, Ocean Currents, Temperature, 
        Salinity</sosKeywords>
      <sosAccessConstraints>NONE</sosAccessConstraints>
      <sosFees>NONE</sosFees>
      
      <!-- Information about the ERDDAP administrator is used for 
      ERDDAP's WMS server. You MUST CHANGE these to describe your 
      installation. -->
      <wmsTitle>NOAA Environmental Research Division 
      WMS</wmsTitle>
      <wmsAbstract>NOAA Environmental Research Division's ERDDAP makes
      data from multiple sources available via the WMS 
      protocol.</wmsAbstract>
      <wmsKeywords>Weather, Ocean Currents, Temperature, 
        Salinity</wmsKeywords> 
      <wmsAccessConstraints>NONE</wmsAccessConstraints>
      <wmsFees>NONE</wmsFees>
      <!-- For the wms examples, pick one of your grid datasets that has 
      longitude and latitude axes. The sample variable must be a variable 
      in the sample grid dataset.  The bounding box values are 
      minx,miny,maxx,maxy.
      -->
      <wmsSampleDatasetID>erdBAssta5day</wmsSampleDatasetID>
      <wmsSampleVariable>sst</wmsSampleVariable>
      <wmsSampleBBox>0,-75,180,75</wmsSampleBBox>
      
      

Changes in version 1.08 (released 2008-07-13)

  • A new web service in ERDDAP, generateDatasetsXml, assists ERDDAP administrators by creating a rough draft of the XML needed to describe a dataset in datasets.xml
  • Some changes/bug fixes related to allowing griddap to be seen by netcdf-java as an opendap server, including: global metadata is now labeled "NC_GLOBAL" (instead of "GLOBAL").
  • The EDDGrid and EDDTable Data Access Forms now utilize query information in the URL. So, for example, if a user goes from a Make A Graph form to a Data Access Form, the constraints are now properly transferred.
  • tabledap's Make A Graph now allows constraints on String variables.
  • EDDTable's Make A Graph now allows NaN constraints. Thanks to Steve Hankin.
  • Bug fix: EDDTable saveAsImage didn't properly recognize the .colorbar min and max values. Thanks to Steve Hankin
  • Many improvements to setupDatasetsXml. Thanks to Ellyn Montgomery.
  • Griddap requests now allow ()-style requests slightly outside of the actual axis range. This is appropriate since ()-values are rounded to the nearest actual value. Thanks to Cindy Bessey
  • I made the FloatArray and DoubleArray test of isEvenlySpaced more sophisticated. It will always be imperfect (because the test would need to be customized for each dataset), but it should be better. Thanks to Ellyn Montgomery.
  • I moved setup.html and setupDatasetsXml.html erddap's /download directory and hard coded all links to them. Now, I can make changes and update the setup information immediately.
  • Many small changes. A few small bug fixes.
  • Things ERDDAP administrators should do to upgrade to this version:
    • Move <theShortDescriptionHtml> from your messages.xml to your setup.xml. It specifies the text that appears in the middle of the left side of the ERDDAP home page. Also, add <h1>ERDDAP</h1> (or some other headline) to the top of it. Or, copy <theShortDescriptionHtml> in the new setup.xml (from the new erddapContent.zip) into your setup.xml.
       

Changes in version 1.06 (released 2008-06-20)

  • New support for IOOS DIF SOS data sources.
  • Many small changes. A few small bug fixes.
     

Changes in version 1.04 (released 2008-06-10)

  • New Slide Sorter feature.
  • New Google Gadgets page and examples.
  • Bug fix in EDDGrid.saveAsNc for variable with scale and addOffset.
     

Changes in version 1.02 (released 2008-05-26)

  • New EDDGridSideBySide allows for different axisVariables[0] sourceValues.
  • All of the currents and winds datasets were merged into EDDGridSideBySide datasets.
  • Images from image requests are now cached for 1 hour.
     

Changes in version 1.00 (released 2008-05-06)

  • Make A Graph web pages and graphics commands in URLs.
  • Support for flag files to force reloading a dataset.
  • New dataset type: EDDTableFrom4DFiles (the first subclass of EDDTableFromFiles).

 

Contact

Questions, comments, suggestions? Please send an email to bob dot simons at noaa dot gov and include the ERDDAP URL directly related to your question or comment.
 

ERDDAP, Version 1.42
Disclaimers | Privacy Policy