NOAA   ERDDAP
Easier access to scientific data

Brought to you by NOAA NMFS SWFSC ERD    
 
Set Up Your Own ERDDAP
 
Why? | Initial Setup | Update | Need To Know | Don't Need To Know |
Slide Shows | Programmer's Guide | List of Changes | Credits | License | Contact

And see the related document Working with the datasets.xml File

ERDDAP is a free and open source, all-Java (servlet), web application that runs in a web application server (for example, Tomcat). This web page is mostly for people ("ERDDAP administrators") who want to set up their own ERDDAP installation at their own web site.

Why use ERDDAP to distribute your data?

Because the small effort to setup ERDDAP brings many benefits.
  • If you already have a web service for distributing your data,
    you can set up ERDDAP to access your data via the existing service.
    Or, you can set up ERDDAP to access your data directly from local files.
  • For each dataset, you only have to write a small chunk of XML to tell ERDDAP how to access the dataset.
  • Once you have ERDDAP serving your data, end users can:
    • Request the data in various ways (DAP, WMS, and more in the future).
    • Get the data response in various file formats. (That's probably the biggest reason!)
    • Make graphs and maps. (Everyone likes pretty pictures.)
You can customize your ERDDAP's appearance so ERDDAP reflects your organization and fits in with the rest of your web site.

ERDDAP has been installed by over 50 organizations worldwide,

including:
  • CSIRO and IMOS (Australia's Commonwealth Scientific and Industrial Research Organisation and the Integrated Marine Observing System)
  • IPSC JRC EC (Maritime Affairs Unit, Institute for the Protection and Security of the Citizen, Joint Research Centre of the European Commission)
  • IRD (Institut de Recherche pour le Développement, France)
    CNRS (Centre National de la Recherche Scientifique, France)
    UPMC (Université Pierre et Marie CURIE, Paris, France)
    UCAD (Université Cheikh Anta Diop de Dakar, Sénégal)
    UGB (Université Gaston Berger - Saint-Louis du Sénégal)
    UFHB (Université Félix HOUPHOUËT-BOIGNY, Abidjan, Côte d'Ivoire)
    IPSL (Institut Pierre Simon Laplace des sciences de l'environnement, Paris, France)
    LMI ECLAIRS (Laboratoire Mixte International «Etude du Climat en Afrique de l’Ouest et de ses Interactions avec l’Environnement Régional, et appui aux services climatiques»)
  • The Marine Institute (Ireland)
  • Marine Instruments S.A. (Spain)
  • NOAA CW CGOM (CoastWatch Caribbean/Gulf of Mexico Node)
  • NOAA CW WC (CoastWatch West Coast Node) which is co-located with and works with
    NOAA ERD (Environmental Research Division of SWFSC of NMFS)
  • NOAA IOOS CeNCOOS (Central and Northern California Ocean Observing System)
  • NOAA IOOS NERACOOS (Northeastern Regional Association of Coastal and Ocean Observing Systems)
  • NOAA IOOS NGDAC (National Glider Data Assembly Center)
  • NOAA IOOS PacIOOS (Pacific Islands Ocean Observing System) at the University of Hawaii (UH)
  • NOAA IOOS SECOORA (Southeast Coastal Ocean Observing Regional Association)
  • NOAA NCEI (National Centers for Environmental Information) / NCDDC
  • NOAA NGDC STP (National Geophysical Data Center, Solar - Terrestrial Physics)
  • NOAA OSMC (Observing System Monitoring Center)
  • NOAA UAF (Unified Access Framework)
  • ONC (Ocean Networks Canada)
  • R.Tech Engineering, France
  • Spyglass Technologies
  • Stanford University, Hopkins Marine Station
  • UC Davis BML (University of California at Davis, Bodega Marine Laboratory)
  • USGS Coastal and Marine Geology Program
  • UW APL (University of Washington, Applied Physics Laboratory)
This is just a list of some of the organizations where ERDDAP has been installed by some individual or some group. It does not imply that the individual, the group, or the organization recommends or endorses ERDDAP.

ERDDAP is recommended within NOAA
NOAA's Data Access Procedural Directive (external link) includes ERDDAP in its list of recommended data servers for use by groups within NOAA.

Is the installation procedure hard? Can I do it?

The initial installation takes some time, but it isn't very hard. You can do it. If you get stuck, email me at bob dot simons at noaa dot gov . I will help you.
Or, you can join the ERDDAP Google Group / Mailing List and post your question there.

How To Do the Initial Setup of ERDDAP on Your Server

ERDDAP can run on any server that supports Java and Tomcat (and perhaps other application servers). ERDDAP has been tested on Linux (including on Amazon's AWS), Mac, and Windows computers.
  • Amazon - If you are installing ERDDAP on an Amazon Web Services EC2 instance, see this Amazon Web Services Overview (below) first.
  • Docker - Axiom now offers ERDDAP in a Docker container (external link).
    If you don't already use Docker, we don't recommend this.
    If you chose to install ERDDAP via Docker, we don't offer any support for the installation process.
    We haven't worked with Docker yet. If you work with this, please send us your comments.
  • Linux and Macs - ERDDAP works great on Linux and Mac computers.
  • Windows - Windows is fine for testing ERDDAP and for personal use, but we don't recommend using it for public ERDDAPs. Running ERDDAP on Windows may have problems: notably, ERDDAP may be unable to delete and/or rename files quickly. This is probably due to antivirus software (e.g., from MacAfee and Norton) which is checking the files for viruses. If you run into this problem (which can be seen by error messages in the log.txt file like "Unable to delete ..."), changing the antivirus software's settings may partially alleviate the problem. Or consider using a Linux or Mac server instead.

The standard ERDDAP installation instructions are:

  1. Set up Java.
    Type "java -version" from your server's command line to make sure you have Java (external link) (JRE or JDK) version 1.8 or higher installed (Java 1.7 is no longer supported because it reached its end of life (external link) and so is no longer supported by Oracle). For security reasons, it is almost always best to use the latest version of Java. ERDDAP works with 32 bit or 64 bit Java. 64 bit is preferred for 64-bit operating systems. On Linux, we recommend that you download and install Java even if your computer came with Java installed. This lets you be in control of which Java you have and where it is (usr/local?). To install Java on Linux, see these instructions (external link).
     
  2. Set up Tomcat (external link).
    For security reasons, it is almost always best to use the latest version of Tomcat. Below, the Tomcat directory will be referred to as tomcat. Warning! If you already have a Tomcat running some other web application (especially THREDDS), we recommend that you install ERDDAP in a second Tomcat, because ERDDAP may need different settings and shouldn't have to contend with other applications for memory.
     
    • Follow the instructions at http://tomcat.apache.org/ (external link) to set up Tomcat on your server.
      On Linux, we recommend installing it in /usr/local.
      On a Mac, it is probably already installed in /Library/Tomcat.
      On Windows, select the "32-bit/64-bit Windows Service Installer". It will be installed in c:/Program Files.
       
    • Security recommendation: you can remove some possible avenues for attackers by following these instructions (external link) (especially the article's second recommendation) to disable certain HTTP methods (TRACE, PUT, OPTIONS, DELETE) in Tomcat. (background information (external link))
       
    • On Linux and Macs, it is best to set up Tomcat (the program) as belonging to user "tomcat" (a separate user with limited permissions and which has no password. Thus, only the super user can switch to acting as user tomcat. This makes it impossible for hackers to log in to your server as user tomcat. And in any case, you should make it so that the tomcat user has very limited permissions on the server's file system (read+write+execute privileges for the apache-tomcat directory tree and <bigParentDirectory> and read-only privileges for directories with data that ERDDAP needs access to).
      • You can create the tomcat user account (which has no password) by using the command
        sudo useradd tomcat -s /bin/bash -p '*'
      • You can switch to working as user tomcat by using the command
        sudo su - tomcat
        (It will ask you for the superuser password for permission to do this.)
      • You can stop working as user tomcat by using the command
        exit
      • Do most of the rest of the Tomcat and ERDDAP setup instructions as user "tomcat". Later, run the startup.sh and shutdown.sh scripts as user "tomcat" so that Tomcat has permission to write to its log files.
      • After unpacking Tomcat, from the parent of the apache-tomcat directory:
        • Change ownership of the apache-tomcat directory tree to the tomcat user.
          chown -R tomcat apache-tomcat-8.0.23
          (but substitute the actual name of your tomcat directory).
        • Change the "group" to be tomcat, your username, or the name of a small group that includes tomcat and all the administrators of Tomcat/ERDDAP, e.g.,
          chgrp -R yourUserName apache-tomcat-8.0.23
        • Change permissions so that tomcat and the group have read, write, execute privileges, e.g,.
          chmod -R ug+rwx apache-tomcat-8.0.23
        • Remove "other" user's permissions to read, write, or execute:
          chmod -R o-rwx apache-tomcat-8.0.23
          This is important, because it prevents other users from reading possibly sensitive information in ERDDAP setup files.

         
    • Set Tomcat's Environmental Variables
      Create a file tomcat/bin/setenv.sh (or \'s and .bat in Windows) (or in Red Hat Enterprise Linux (RHEL), edit ~tomcat/conf/tomcat8.conf) to set Tomcat's environmental variables. This file will be used by tomcat/bin/startup.sh and shutdown.sh (or \'s and .bat in Windows). The file should contain:
      export JAVA_HOME=/usr/local/jdk1.8.0_77/jre
      export JAVA_OPTS='-server -Djava.awt.headless=true -Xmx1500M -Xms1500M'
      export TOMCAT_HOME=/usr/local/apache-tomcat-8.0.23
      export CATALINA_HOME=/usr/local/apache-tomcat-8.0.23

      (but substitute the directory names from your computer).
      (If you previously set JRE_HOME, you can remove that.)
      In Windows, use set instead of export and \'s instead of /'s.
      On Macs, you probably don't need to set JAVA_HOME.
      The -Xmx and -Xms memory settings are important because ERDDAP works better with more memory. Always set -Xms to the same value as -Xmx.
      • For 32 bit Operating Systems and 32 bit Java:
        The more physical memory in the server the better: 4+ GB is really good, 2 GB is okay, less is not recommended. With 32 bit Java, even with abundant physical memory, Tomcat and Java won't run if you try to set -Xmx much above 1500M (1200M on some computers). If your server has less than 2GB of memory, reduce the -Xmx value (in 'M'egaBytes) to 1/2 of the computer's physical memory.
      • For 64 bit Operating Systems and 64 bit Java:
        64 bit Java will only work on a 64 bit operating system. To enable 64 bit Java, add -d64 to the list of JAVA_OPTS in startup.sh and shutdown.sh, for example,
        export JAVA_OPTS='-server -Djava.awt.headless=true -Xmx8000M -Xms8000M -d64'
        With 64 bit Java, Tomcat and Java can use very high -Xmx and -Xms settings. The more physical memory in the server the better. As a simplistic suggestion: we recommend you set -Xmx and -Xms to (in 'M'egaBytes) to 1/2 (or less) of the computer's physical memory. You can see if Tomcat, Java, and ERDDAP are indeed running in 64 bit mode by searching for " bit," in ERDDAP's Daily Report email or in the bigParentDirectory/logs/log.txt file (bigParentDirectory is specified in setup.xml).
         
    • On Linux and Macs, change the permissions of all *.sh files in tomcat/bin/ to be executable by the owner, e.g., with
      chmod +x *.sh
       
    • Fonts for images: We strongly prefer the free Vera Sans fonts to the standard Linux/Java fonts. Installing these fonts isn't required.

      If you don't install these fonts, you need to change the fontFamily setting in setup.xml to <fontFamily>SansSerif</fontFamily> .

      To install the fonts, please download BitstreamVeraSans.zip (344,753 bytes, MD5=E16AF0C9838FD2443434F6E0E9FD0A0D) and unzip the font files to a temporary directory.

      • On Linux (as the root user) and Windows XP (as the administrator), copy the font files into JAVA_HOME/lib/fonts so Java can find the fonts. Remember: if/when you later upgrade to a newer version of Java, you need to reinstall these fonts.
      • On Macs, for each font file, double click on it and then click Install Font.
      • On Windows Vista and 7, in Windows Explorer, select all of the font files. Right click. Click on Install.
         
    • Test your Tomcat installation.
      • Linux:
        As user "tomcat", run tomcat/bin/startup.sh
        View your URL + ":8080/" in your browser (e.g., http://coastwatch.pfeg.noaa.gov:8080/).
        You should see the Tomcat "Congratulations" page.
        If there is trouble, see the Tomcat log file tomcat/logs/catalina.out.
      • Mac (run tomcat as the system administrator user):
        Run tomcat/bin/startup.sh
        View your URL + ":8080/" in your browser (e.g., http://coastwatch.pfeg.noaa.gov:8080/).
        You should see the Tomcat "Congratulations" page.
        If there is trouble, see the Tomcat log file tomcat/logs/catalina.out.
      • Windows localhost:
        Right click on the Tomcat icon in the system tray, and choose "Start service".
        View http://127.0.0.1:8080/ in your browser.
        You should see the Tomcat "Congratulations" page.
        If there is trouble, see the Tomcat log file tomcat/logs/catalina.out.
         
    • Troubles with the Tomcat installation?
      • On Linux and Mac, if you can't reach Tomcat or ERDDAP (or perhaps you just can't reach them from a computer outside your firewall), you can test if Tomcat is listening to port 8080, by typing (as root) on a command line of the server:
        netstat -tuplen | grep 8080
        That should return one line with something like:
        tcp 0 0 :::8080 :::* LISTEN ## ##### ####/java
        (where '#' is some digit), indicating that a "java" process (presumably Tomcat) is listening on port "8080" for "tcp" traffic. If no lines were returned, if the line returned is significantly different, or if two or more lines were returned, then there may be a problem with the port settings.
      • See the Tomcat log file tomcat/logs/catalina.out. Tomcat problems and some ERDDAP startup problems are almost always indicated there. This is common when you are first setting up ERDDAP.
      • See the Tomcat (external link) web site or search the web for help, but please let us know the problems you had and the solutions you found.
      • Email me at bob dot simons at noaa dot gov . I will try to help you.
      • Or, you can join the ERDDAP Google Group / Mailing List and post your question there.
         
  3. Set up the tomcat/content/erddap configuration files.
    On Linux, Mac, and Windows, download erddapContent.zip (version 1.72, size=26,760 bytes, MD5=8EFBD16CEBC249A2E21CFE167423635B, dated 2016-05-12) and unzip it into tomcat, creating tomcat/content/erddap .

    For Red Hat Enterprise Linux (RHEL), unzip it into ~tomcat and set the system property erddapContentDirectory=~tomcat/content/erddap in ~tomcat/conf/tomcat8.conf so ERDDAP can find the directory.

    [Some previous versions are also available:
    1.58 (size=25,000 bytes, MD5=620A54B6A52D7C95C0D1635A9FF39DC6, dated 2015-02-25)
    1.60 (size=25,035 bytes, MD5=10332FC959B72F17E6A00AD432A38578, dated 2015-03-12)
    1.62 (size=25,035 bytes, MD5=4B17A93CE52087C53AF6AEA905391418, dated 2015-06-08)
    1.64 (size=24,996 bytes, MD5=8E375D13D388E0D290110E196117E3DC, dated 2015-08-19)
    1.66 (size=25,887 bytes, MD5=70B46B5E96DD17F4DCEAC27CC95BDFBB, dated 2016-01-19)
    1.68 (size=25,896 bytes, MD5=8E375D13D388E0D290110E196117E3DC, dated 2016-02-08)
    1.70 (size=26,759 bytes, MD5=97084B0ABE38D4DE88408918AD301AD9, dated 2016-04-15) ]

    Then,

    • Read the comments in tomcat/content/erddap/setup.xml and make the requested changes. setup.xml is the file with all of the settings which specify how your ERDDAP behaves.
      For the initial setup, you MUST at least change these settings: <bigParentDirectory>, <emailEverythingTo>, <baseUrl>, <baseHttpsUrl>, <email.*>, <admin.*> (and <fontFamily> if you are using Bitstream Vera Sans fonts).

      When you create the bigParentDirectory, from the parent directory of bigParentDirectory:

      • Make user=tomcat the owner of the bigParentDirectory, e.g.,
        chown -R tomcat bigParentDirectory
      • Change the "group" to be tomcat, your username, or the name of a small group that includes tomcat and all the administrators of Tomcat/ERDDAP, e.g.,
        chgrp -R yourUserName bigParentDirectory
      • Change permissions so that tomcat and the group have read, write, execute privileges, e.g,.
        chmod -R ug+rwx bigParentDirectory
      • Remove "other" user's permissions to read, write, or execute:
        chmod -R o-rwx bigParentDirectory reading possibly sensitive information in ERDDAP log files and files with information about private datasets.
    • Read the comments in Working with the datasets.xml File, then modify the XML in tomcat/content/erddap/datasets.xml to specify all of the datasets you want your ERDDAP to serve.
    • (Unlikely) Now or (slightly more likely) in the future, if you want to modify erddap's CSS file, make a copy of tomcat/content/erddap/images/erddapStart.css called erddap.css and then make changes to it. Changes to erddap.css only take affect when ERDDAP is restarted and often also require the user to clear the browser's cached files.
    After you edit the .xml files, it is a good idea to verify that the result is well-formed XML by pasting the XML text into an XML checker like RUWF (external link).

    In the unusual situation where you aren't allowed to modify the Tomcat directory, you can put the ERDDAP content directory somewhere else (e.g., /usr/local/erddap). To let ERDDAP know where it is, set the system property erddapContentDirectory=/usr/local/erddap/content/erddap/ (or wherever it is). If you aren't allowed to set this property in startup.sh, perhaps you can set it in Tomcat's context.xml.

  4. Install the erddap.war file.
    On Linux, Mac, and Windows, download erddap.war (version 1.72, size=527,743,058 bytes, MD5=4C84507540C6868413E6CEE470414665, dated 2016-05-12) into tomcat/webapps . The .war file is big because it contains high resolution coastline, boundary, and elevation data needed to create maps.

    [Some previous versions are also available. If you download one of these, rename it erddap.war.
    1.58 (size=490,902,857 bytes, MD5=B4D1BBDA2FADD88FDCA29461C2578D3D, dated 2015-02-25)
    1.60 (size=490,919,033 bytes, MD5=99319916DA726030C25E0045A65B0971, dated 2015-03-12)
    1.62 (size=491,038,285 bytes, MD5=1FDF3F349EC7D87A4A88AD14A7B5F41A, dated 2015-06-09)
    1.64 (size=511,898,199 bytes, MD5=0E11741BF06EE5388C022F3632EE1915, dated 2015-08-19)
    1.66 (size=512,007,729 bytes, MD5=71CB248B05F0340BE6CF4CF71F3BD8E6, dated 2016-01-19)
    1.68 (size=511,995,047 bytes, MD5=0E919CBDAD014B724DE89F40C8CA9379, dated 2016-02-08)
    1.70 (size=556,163,762 bytes, MD5=3320ADB87E9F3AB31EBF89260C64AF19, dated 2016-04-15) ]

  5. Use ProxyPass so users don't have to put the port number, e.g., :8080, in the URL.
    On Linux computers, if Tomcat is running in Apache, please modify the
    /etc/httpd/conf/httpd.conf file to allow HTTP traffic to/from ERDDAP without requiring the port number, e.g., :8080, in the URL:
    1. Modify the existing <VirtualHost> tag (if there is one), or add one at the end of the file:
      <VirtualHost *:80>
         ServerName YourDomain.org
         ProxyRequests Off
         ProxyPreserveHost On
         ProxyPass /erddap http://localhost:8080/erddap
         ProxyPassReverse /erddap http://localhost:8080/erddap
      </VirtualHost>
      
    2. Then restart Apache: /usr/sbin/apachectl restart (but sometimes it is in a different directory).
       
  6. Start Tomcat.
    (I don't recommend using the Tomcat Web Application Manager. If you don't fully shutdown and startup Tomcat, sooner or later you will have PermGen memory issues.)
    • If Tomcat is already running, use tomcat/bin/shutdown.sh (or \'s and .bat in Windows) to shut down Tomcat.
      On Linux, you may want to use ps -ef | grep tomcat to see if/when the process has been stopped. (It may take a minute or two.)
    • Use tomcat/bin/startup.sh (or \'s and .bat in Windows) to start Tomcat.
       
  7. Is ERDDAP running?
    Use a browser to try to view http://www.YourServer.org/erddap/
    ERDDAP starts up without any datasets loaded. Datasets are loaded in a background thread and so become available one-by-one.

    Trouble? Look in the log files.
    If ERDDAP won't run or some datasets won't load, look for error messages in
    Tomcat's log file (tomcat/logs/catalina.out) and
    ERDDAP's log file (bigParentDirectory/logs/log.txt).

    Trouble: Old Version of Java
    If you are using a version of Java that is too old for ERDDAP, ERDDAP won't run and you will see an error message in Tomcat's log file like
    Exception in thread "main" java.lang.UnsupportedClassVersionError:
    some/class/name: Unsupported major.minor version someNumber

    The solution is to update to the most recent version of Java and make sure that Tomcat is using it.

    Trouble: Slow Startup First Time
    Tomcat has to do a lot of work the first time an application like ERDDAP is started. On some servers, the first attempt to view ERDDAP stalls (30 seconds?) until this work is finished. On other servers, the first attempt will fail immediately. But if you wait 30 seconds and try again, it will succeed if ERDDAP was installed correctly.
    There is no fix for this, but it usually only occurs the first time after you install a new version of ERDDAP.

  8. In the future, to shut down (and restart) ERDDAP, see
    How to Shut Down and Restart Tomcat and ERDDAP.
     
  9. Troubles installing Tomcat or ERDDAP?
    Email me at bob dot simons at noaa dot gov . I will help you.
    Or, you can join the ERDDAP Google Group / Mailing List and post your question there.
     
  10. Email Notification of New Versions of ERDDAP
    If you want to receive an email whenever a new version of ERDDAP is available, send an email to bob dot simons at noaa dot gov requesting to be added to the ERDDAP Announcements mailing list. This list averages roughly one email every three months.
     
  11. Customize your ERDDAP to highlight your organization (not ERD).
    • To change the banner that appears at the top of all ERDDAP .html pages, change the HTML content specified in the setup.xml file in the <startBodyHtml> tag. For example, you could:
      • Use a different image (i.e., your organization's logo).
      • Change the background color.
      • Change "ERDDAP" to "YourOrganization's ERDDAP"
      • Change "Easier access to scientific data" to "Easier access to YourOrganization's data".
      • Change the "Brought to you by" links to be links to your organization and funding sources.
    • To change the information on the left side of the home page, e.g., to information about your project, change the HTML content specified in the setup.xml file in the <theShortDescriptionHtml> tag.
    • To change the icon that appears on browser tabs, put your organization's favicon.ico in tomcat/content/erddap/images/ . See https://en.wikipedia.org/wiki/Favicon (external link).

How To Do an Update of an Existing ERDDAP on Your Server

  1. Make the changes listed in Changes in the section entitled "Things ERDDAP Administrators Need to Know and Do" for all of the ERDDAP versions since the version you were using.
     
  2. Download erddap.war (version 1.72, size=527,743,058 bytes, MD5=4C84507540C6868413E6CEE470414665, dated 2016-05-12) into a temporary directory.
     
  3. messages.xml
    • Common: If you are upgrading from ERDDAP version 1.46 (or above) and you just use the standard messages, the new standard messages.xml will be installed automatically (amongst the .class files via erddap.war).
       
    • Rare: If you are upgrading from ERDDAP version 1.44 (or below),
      you MUST delete the old messages.xml file:
      tomcat/content/erddap/messages.xml .
      The new standard messages.xml will be installed automatically (amongst the .class files via erddap.war).
       
    • Rare: If you always make changes to the standard messages,
      you need to make those changes to the new messages.xml file (which is
      WEB-INF/classes/gov/noaa/pfel/erddap/util/messages.xml after erddap.war is decompressed by Tomcat).
       
    • Rare: If you maintain a custom messages.xml file
      in tomcat/content/erddap/, you need to figure out (via diff) what changes have been made to the default messages.xml (which are in the new erddap.war as
      WEB-INF/classes/gov/noaa/pfel/erddap/util/messages.xml) and modify your custom messages.xml file accordingly.
       
  4. Install the new ERDDAP in Tomcat:
    1. (Don't use Tomcat Manager. Sooner or later there will be PermGen memory issues. It is better to actually shutdown and startup Tomcat.)
    2. From a command line, use tomcat/bin/shutdown.sh (or \'s and .bat in Windows).
      On Linux, you may want to use ps -ef | grep tomcat to see if/when the process has been stopped. (It may take a minute or two.)
    3. In tomcat/webapps, use rm -r erddap
    4. In tomcat/webapps, use rm erddap.war
    5. Copy the new erddap.war file from the temporary directory to tomcat/webapps .
    6. Use tomcat/bin/startup.sh (or \'s and .bat in Windows) to restart Tomcat and ERDDAP.
    7. View ERDDAP in your browser to check that the restart succeeded. (Sometimes, you need to try to load ERDDAP in your browser a few times before it succeeds.)
       
  5. Troubles updating ERDDAP?
    Email me at bob dot simons at noaa dot gov . I will help you.
    Or, you can join the ERDDAP Google Group / Mailing List and post your question there.

Things You Need To Know

  • Use Ctrl-F To Find Things On This Web Page
    All of the information about administering ERDDAP (other than working with datasets.xml) is on this one, very long, .html web page, not several .html pages as some people prefer. The advantage of one .html web page is that you can use Ctrl-F (Command-F on a Mac) in your web browser to search for text (for example, flag) within this web page.

    Alternatively, at the top of this document, there is a list of main topics (a Table of Contents).

  • If ERDDAP Is Responding Slowly
    you may be able to figure out if the slowness is reasonable and temporary (e.g., because of lots of requests from scripts or WMS users), or if ERDDAP is having trouble and you need to shut down and restart Tomcat and ERDDAP.

    If ERDDAP is responding slowly, here are some things you can do to determine the cause:

    • Look for clues (e.g., error messages) in
      ERDDAP's log file (bigParentDirectory/logs/log.txt) and
      Tomcat's log file (tomcat/logs/catalina.out).
       
    • If just one dataset is responding slowly (perhaps only for certain types of requests), the problem may be:
      • ERDDAP's access to the dataset's source data may be temporarily or permanently slow. Try to check the source's speed independent of ERDDAP. If it is slow, perhaps you can improve it.
      • The dataset may be not optimally configured. You may be able to make changes to the dataset's datasets.xml chunk to help ERDDAP handle the dataset better. For example, the configuration of the <subsetVariables> tag has a huge influence on how ERDDAP handles EDDTable datasets. Similarly, many factors influence the speed of an EDDTableFromDatabase dataset. If you want help speeding up a specific dataset, email a description of the problem and the dataset's chunk of datasets.xml to bob dot simons at noaa dot gov.
        Or, you can join the ERDDAP Google Group / Mailing List and post your question there.
         
    • If everything in ERDDAP is always slow, the problem may be:
      • The computer that is runnning ERDDAP may not have enough memory or processing power. It is good to run ERDDAP on a modern, multi-core server. For heavy use, the server should have a 64-bit operating system and 8 GB or more of memory.
      • The computer that is runnning ERDDAP may be also running other applications that are consuming lots of system resources. If so, can you get a dedicated server for ERDDAP? For example (this is not an endorsement), you can get a quad-core Mac Mini Server with 8 GB of memory for ~$1100.
         
    • If everything in ERDDAP is temporarily slow, view your ERDDAP's
      /erddap/status.html page in your browser.
      • Does the ERDDAP status page fail to load?
        Is so, restart ERDDAP.
      • Did the ERDDAP status page load slowly (e.g., >5 seconds)?
        That is a sign that everything in ERDDAP is running slowly, but it isn't necessarily trouble.
      • For "Response Failed Time (since last major LoadDatasets)", is n= a large number?
        That indicates there have been lots of failed requests recently. That may be trouble or the start of trouble. The median time for the failures is often large (e.g., 210000 ms),
        which means that there were (are?) lots of active threads.
        which were tying up lots of resources (like memory, open files, open sockets, ...),
        which is not good.
      • For "Response Succeeded Time (since last major LoadDatasets)", is n= a large number?
        That indicates there have been lots of successful requests recently. This isn't trouble. It just means your ERDDAP is getting heavy use.
      • Is the "Number of non-Tomcat-waiting threads" double a typical value?
        This is often serious trouble that will cause ERDDAP to slow down and eventually freeze. If this persists for hours, you may want to proactively restart ERDDAP.
      • At the bottom of the "Memory Use Summary" list, is the last "Memory: currently using" value very high?
        That may just indicate high usage, or it may be a sign of trouble.
         
    • Is your internet connection currently slow?
      Search the internet for "internet speed test" and use one of the free online tests, such as http://www.speakeasy.net/speedtest// (external link). If your internet connection is slow, it slows down ERDDAP. Sometimes, you can solve this by stopping unnecessary internet use (e.g., people watching streaming videos or on video conference calls).
       
    • Use the "Server Status" link in your Tomcat Manager to check on the status of all of the response threads.
       
  • How to Shut Down and Restart Tomcat and ERDDAP
    You don't need to shut down and restart Tomcat and ERDDAP if ERDDAP is temporarily slow, or slow for some known reason (like lots of requests from scripts or WMS users).

    You do need to shut down and restart Tomcat and ERDDAP if you need to apply changes to the setup.xml file, or if ERDDAP freezes, hangs, or locks up.

    To shutdown and restart Tomcat and ERDDAP:  

    • If you use Linux or a Mac:
      1. Use cd tomcat/bin
      2. Use ps -u tomcatUser | grep java to find the java/tomcat processID (hopefully, just one process will be listed), which we'll call javaProcessID below.
      3. If ERDDAP is frozen/hung/locked up, use kill -3 javaProcessID to tell Java (which is running Tomcat) to do a thread dump to the Tomcat log file
        tomcat/logs/catalina.out. After you reboot, you can diagnose the problem by finding the thread dump information (and any other useful information above it) in tomcat/logs/catalina.out and also by reading relevant parts of the ERDDAP log archive. If you want, you can email that information to bob dot simons at noaa dot gov so I can see what went wrong.
        Or, you can join the ERDDAP Google Group / Mailing List and post your question there.
      4. Use ./shutdown.sh
      5. Use ps -u tomcatUser | grep java repeatedly until the java/tomcat process isn't listed.
        Sometimes, the java/tomcat process will take up to two minutes to shut down. The reason is: ERDDAP sends a message to its background threads to tell them to stop, but sometimes it takes these threads up to two minutes to get to a good stopping place.
      6. If you don't want to wait for java/tomcat to stop by itself, you can use
        kill -9 javaProcessID
        to force the java/tomcat process to stop immediately. If possible, use this only as a last resort. The -9 switch is powerful, but it may cause various problems.
      7. To restart ERDDAP, use ./startup.sh
      8. View ERDDAP in your browser to check that the restart succeeded. (Sometimes, you need to wait 30 seconds and try to load ERDDAP again in your browser for it to succeed.)
         
    • If you use Windows:
      1. Use cd tomcat/bin
      2. Use shutdown
      3. You may want/need to use the Windows Task Manager (accessible via Ctrl Alt Del) to ensure that the Java/Tomcat/ERDDAP process/application has fully stopped.
        Sometimes, the process/application will take up to two minutes to shut down. The reason is: ERDDAP sends a message to its background threads to tell them to stop, but sometimes it takes these threads up to two minutes to get to a good stopping place.
      4. To restart ERDDAP, use startup
      5. View ERDDAP in your browser to check that the restart succeeded. (Sometimes, you need to wait 30 seconds and try to load ERDDAP again in your browser for it to succeed.)
         
  • Frequent Crashes - If ERDDAP freezes/stops twice or more in one day, you probably have an troublesome user running several scripts at once and/or someone making a large number of invalid requests. If this happens, you should probably blacklist that user. When a blacklisted user makes a request, the error message in the response encourages them to email you to work out the problems. Then, you can encourage them to run just one script at a time and to fix the problems in their script (e.g., requesting data from a remote dataset that can't respond before timing out). See <requestBlacklist> in your datasets.xml file.
     
  • Monitor ERDDAP - My experience is that ERDDAP can run for months without needing a restart. But sometimes, if there is heavy use (e.g., scripts or WMS users) and especially if many of the requests are failing in odd ways, then ERDDAP freezes (and needs to be restarted) every few days or even every day. I suspect that the problem is related to the program stopping lower level processing threads that froze or otherwise failed in an odd way. Java only offers one way for a program to stop these threads and that method is not good (and deprecated) because it is highly likely that it causes memory and other resource leaks. Those leaks accumulate. After 1000's of leaks, ERDDAP becomes very slow or freezes. Unfortunately, I haven't found which resource leak is causing the problem. But if you look at your ERDDAP's /erddap/status.html page, when the number on the line
    Number of non-Tomcat-waiting threads in this JVM
    increases significantly (e.g., doubles) and permanently (more than an hour or two), that is usually a sign that trouble is here or is coming soon.

    If you find that you frequently need to restart ERDDAP, you might try using Monit (external link) to monitor your ERDDAP and restart it if needed. Or, you could make a cron job to restart ERDDAP (proactively) periodically. It may be a little challenging to write a script to automate monitoring and restarting ERDDAP. Some tips that might help:

    • You can simplify testing if the Tomcat process is still running by using the -c switch with grep:
      ps -u tomcatUser | grep -c java
      That will reduce the output to "1" if the tomcat process is still alive, or "0" if the process has stopped.
       
    • If you are good with gawk, you can extract the processID from the results of
      ps -u tomcatUser | grep java, and use the processID in other lines of the script.
       
    If you do set up Monit or a cron job, please email the details to bob dot simons at noaa dot gov .
    Or, you can join the ERDDAP Google Group / Mailing List and share the information there.
     
  • PermGen - If you repeatedly use Tomcat Manager to Reload (or Stop and Start) ERDDAP, ERDDAP may fail to start up and throw java.lang.OutOfMemoryError: PermGen. The solution is to periodically (or every time?) shut down and restart tomcat and ERDDAP, instead of just reloading ERDDAP.
    [Update: This problem should be greatly minimized or fixed in ERDDAP version 1.24.]
     
  • log.txt - If ERDDAP doesn't start up or if something isn't working as expected, it is very useful to look at the error and diagnostic messages in the ERDDAP log file.
    • The log file is bigParentDirectory/logs/log.txt
      (bigParentDirectory is specified in setup.xml). If there is no log.txt file or if the log.txt file hasn't been updated since you restarted ERDDAP, look in the Tomcat Log Files to see if there is an error message there.
    • Types of diagnostic messages in the log file:
      • The word "error" is used when something went so wrong that the procedure failed to complete. Although it is annoying to get an error, the error forces you to deal with the problem. Our thinking is that it is better to throw an error, than to have ERDDAP hobble along, working in a way you didn't expect.
      • The word "warning" is used when something went wrong, but the procedure was able to complete. These are pretty rare.
      • Anything else is just an informative message. You can control how much information is logged with <logLevel> in setup.xml.
    • Information is written to the log file on the disk drive in fairly big chunks. The advantage is that this is very efficient -- ERDDAP will never block waiting for information to be written to the log file. The disadvantage is that the log will almost always end with a partial message, which won't be completed until the next chunk is written. You can make it up-to-date (for an instant) by viewing your ERDDAP's status web page at http://your.domain.org/erddap/status.html .
    • When the log.txt files gets to 20 MB,
      the file is renamed log.txt.previous and a new log.txt file is created. So log files don't accumulate.

      In setup.xml, you can specify a different maximum size for the log file, in MegaBytes. The minimum allowed is 1 (MB). The maximum allowed is 2000 (MB). The default is 20 (MB). For example:
      <logMaxSizeMB>20</logMaxSizeMB>

    • Whenever you restart ERDDAP,
      ERDDAP makes an archive copy of the log.txt and log.txt.previous files with a time stamp in the file's name. If there was trouble before the restart, it may be useful to analyze these archived files for clues as to what the trouble was. You can delete the archive files if they are no longer needed.
       
  • Tomcat Log Files - If ERDDAP doesn't start up because an error occured very early in ERDDAP's startup, the error message will show up in Tomcat's log files (tomcat/logs/catalina.today.log or tomcat/logs/catalina.out), not in ERDDAP's log.txt file.
     
  • emailLogYEAR-MM-DD.txt - ERDDAP always writes the text of all out-going email messages in the current day's emailLogYEAR-MM-DD.txt file in bigParentDirectory/logs (bigParentDirectory is specified in setup.xml).
    • If the server can't send out email messages, or if you have configured ERDDAP not to send out email messages, or if you are just curious, this file is a convenient way to see all of the email messages that have been sent out.
    • You can delete previous days' email log files if they are no longer needed.
       
  • Daily Report - The Daily Report has lots of useful information -- all of the information from your ERDDAP's /erddap/status.html page and more.
    • It is the most complete summary of your ERDDAP's status.
    • Among other statistics, it includes a list of datasets that didn't load and the exceptions they generated.
    • It is generated when you start up ERDDAP (just after ERDDAP finishes trying to load all of the datasets) and generated soon after 7 am local time every morning.
    • Whenever it is generated, it is written to ERDDAP's log.txt file.
    • Whenever it is generated, it is emailed to <emailDailyReportsTo> and <emailEverythingTo> (which are specified in setup.xml) provided you have set up the email system (in setup.xml).
       
  • Status Page - You can view the status of your ERDDAP from any browser by going to <baseUrl>/erddap/status.html
    • This page is generated dynamically, so it always has up-to-the-moment statistics for your ERDDAP.
    • It includes statistics regarding the number of requests, memory usage, thread stack traces, the taskThread, etc.
    • Because the Status Page can be viewed by anyone, it doesn't include quite as much information as the Daily Report.
       
  • Adding/Changing Datasets - ERDDAP usually rereads datasets.xml every loadDatasetsMinMinutes (specified in setup.xml). So you can make changes to datasets.xml any time, even while ERDDAP is running.
    A new dataset will be detected soon, usually within loadDatasetsMinMinutes.
    A changed dataset will be reloaded when it is reloadEveryNMinutes old (as specified in datasets.xml).
     
  • A Flag File Tells ERDDAP to Try to Reload a Dataset As Soon As Possible
    • ERDDAP won't notice any changes to a dataset's setup in datasets.xml until ERDDAP reloads the dataset.
    • If a dataset is active in ERDDAP and you want to force ERDDAP to reload it as soon as possible (before the dataset's <reloadEveryNMinutes> would cause it to be reloaded), put a file in bigParentDirectory/flag (bigParentDirectory is specified in setup.xml) that has the same name as the dataset's datasetID.
    • The contents of the flag file are irrelevant.
    • In between major dataset reloads, ERDDAP looks continuously for flag files.
    • When ERDDAP finds a flag file, it deletes the file and tries to reload the dataset very soon (usually within seconds).
    • Note that when a dataset is reloaded, all files in the bigParentDirectory/cache/datasetID directory are deleted.
    • Note that if the dataset's xml includes active="false", a flag will cause the dataset to be made inactive (if it is active), and in any case, not reloaded.

    ERDDAP has a web service so that flags can be set via URLs.

    • For example,
      https://coastwatch.pfeg.noaa.gov/erddap/setDatasetFlag.txt?datasetID=rPmelTao&flagKey=123456789
      (that's a fake flagKey) will set a flag for the rPmelTao dataset.
    • There is a different flagKey for each datasetID.
    • Administrators can see a list of flag URLs for all datasets by looking at the bottom of their Daily Report email.
    • Administrators should treat these URLs as confidential, since they give someone the right to reset a dataset at will.
    • If you think the flagKeys have fallen into the hands of someone who is abusing them, you can change <flagKeyKey> in setup.xml and restart ERDDAP to force ERDDAP to generate and use a different set of flagKeys.
    • If you change <flagKeyKey>, delete all of the old subscriptions (see the list in your Daily Report) and remember to send the new URLs to the people who you do want to have them.

    The flag system can serve as the basis for a more efficient mechanism for telling ERDDAP when to reload a dataset. For example, you could set a dataset's <reloadEveryNMinutes> to a large number (e.g., 10080 = 1 week). Then, when you know the dataset has changed (perhaps because you added a file to the dataset's data directory), set a flag so that the dataset is reloaded as soon as possible. Flags are usually seen quickly. But if the LoadDatasets thread is already busy, it may be a while before it is available to act on the flag. But the flag system is much more responsive and much more efficient than setting <reloadEveryNMinutes> to a small number.

  • Force Dataset Removal - If a dataset is active in ERDDAP, and you want to deactivate it as soon as possible, add active="false" to the dataset tag and set a flag. Flags are usually seen quickly. But if the LoadDatasets thread is already busy, it may be a while before it is available to act on the flag. Once the dataset is not active (i.e., not visible in ERDDAP's list of datasets), you can remove the dataset's description from the datasets.xml file if you want to.
     
  • When Are Datasets Reloaded? A thread called RunLoadDatasets is the master thread that controls when datasets are reloaded. RunLoadDatasets loops forever:
    1. RunLoadDatasets notes the current time.
    2. RunLoadDatasets starts a LoadDatasets thread to do a "majorLoad". You can see information about the current/previous majorLoad at the top of your ERDDAP's
      /erddap/status.html page (status page example).
      1. LoadDatasets makes a copy of datasets.xml.
      2. LoadDatasets reads through the copy of datasets.xml and, for each dataset, sees if the dataset needs to be (re)loaded or removed.
        • If a flag file exists for this dataset, the file is deleted and the dataset is removed if active="false" or (re)loaded if active="true" (regardless of the dataset's age).
        • If the dataset's dataset.xml chunk has active="false" and the dataset is currently loaded (active), is is unloaded (removed).
        • If the dataset has active="true" and the dataset isn't already loaded, it is loaded.
        • If the dataset has active="true" and the dataset is already loaded, the data set is reloaded if the dataset's age (time since last load) is greater than its <reloadEveryNMinutes> (default = 10080 minutes), otherwise, the dataset is left alone.
      3. LoadDatasets finishes.
      The RunLoadDatasets thread waits for the LoadDatasets thread to finish. If LoadDatasets takes longer than loadDatasetsMinMinutes (as specified in setup.xml), RunLoadDatasets interrupts the LoadDatasets thread. Ideally, LoadDatasets notices the interrupt and finishes. But if it doesn't notice the interrupt within a minute, RunLoadDatasets calls loadDatasets.stop(), which is undesirable.
    3. While the time since the start of the last majorLoad is less than loadDatasetsMinMinutes (as specified in setup.xml, e.g., 15 minutes), RunLoadDatasets repeatedly looks for flag files in the bigParentDirectory/flag directory. If one or more flag files are found, they are deleted, and RunLoadDatasets starts a LoadDatasets thread to do a "minorLoad" (majorLoad=false). You can't see minorLoad information on your ERDDAP's /erddap/status.html page.
      1. LoadDatasets makes a copy of datasets.xml.
      2. LoadDatasets reads through the copy of datasets.xml and, for each dataset for which there was a flag file:
        • If the dataset's dataset.xml chunk has active="false" and the dataset is currently loaded (active), is is unloaded (removed).
        • If the dataset has active="true", the dataset is (re)loaded, regardless of its age.
        Non-flagged datasets are ignored.
      3. LoadDatasets finishes.
    4. RunLoadDatasets goes back to step 1.

    Notes:
    • Startup
      When you restart ERDDAP, every dataset with active="true" is loaded.
    • Cache
      When a dataset is (re)loaded, its cache (including any data response files and/or image files) is emptied.
    • Lots of Datasets
      If you have a lot of datasets and/or one or more datasets are slow to (re)load, a LoadDatasets thread may take a long to finish its work, perhaps even longer than loadDatasetsMinMinutes.
    • One LoadDatasets Thread
      There is never more than one LoadDatasets thread running at once. If a flag is set when LoadDatasets is already running, the flag probably won't be noticed or acted on until that LoadDatasets thread finishes running. You might say: "That's stupid. Why don't you just start a bunch of new threads to load datasets?" But if you have lots of datasets which get data from one remote server, even one LoadDatasets thread will put substantial stress on the remote server. The same is true if you have lots of datasets which get data from files on one RAID. There are rapidly diminishing returns from having more than one LoadDatasets thread.
    • Flag = ASAP
      Setting a flag just signals that the dataset should be (re)loaded as soon as possible, not necessarily immediately. If no LoadDatasets thread is currently running, the dataset will start to be reloaded within a few seconds. But if a LoadDatasets thread is currently running, the dataset probably won't be reloaded until after that LoadDatasets thread is finished.
    • Flag File Deleted
      In general, if you put a flag file in the bigParentDirectory/erddap/flag directory (by visiting the dataset's flagUrl or putting an actual file there), the dataset will usually be reloaded very soon after that flag file is deleted.
    • Flag vs. Small reloadEveryNMinutes
      If you have some external way of knowing when a dataset needs to be reloaded and if it is convenient for you, the best way to make sure that a dataset is always up-to-date is to set its reloadEveryNMinutes to a large number (10080?) and set a flag (via a script?) whenever it needs to be reloaded. That is the system that EDDGridFromErddap and EDDTableFromErddap use receive messages that the dataset needs to be reloaded.
    • Look in log.txt
      Lots of relevant information is written to the bigParentDirectory/logs/log.txt file. If things aren't working as you expect, looking at log.txt lets you diagnose the problem by finding out exactly what ERDDAP did.
      • Search for "majorLoad=true" for the start of major LoadDataset threads.
      • Search for "majorLoad=false" for the start of minor LoadDatasets threads.
      • Search for a given dataset's datasetID for information about it being (re)loaded or queried.

       
  • Cached Responses - In general, ERDDAP doesn't cache (store) responses to user requests. The rationale was that most requests would be slightly different so the cache wouldn't be very effective. The biggest exceptions are requests for image files (which are cached since browsers and programs like Google Earth often re-request images) and requests for .nc files (because they can't be created on-the-fly). ERDDAP stores each dataset's cached files in a different directory: bigParentDirectory/cache/datasetID since a single cache directory might have a huge number of files which might become slow to access.
    Files are removed from the cache for one of three reasons:
    • All files in this cache are deleted when ERDDAP is restarted.
    • Periodically, any file more than <cacheMinutes> old (as specified in setup.xml) will be deleted. Removing files in the cache based on age (not Least-Recently-Used) ensures that files won't stay in the cache very long. Although it might seem like a given request should always return the same response, that isn't true. For example, a tabledap request which includes &time>someTime will change if new data arrives for the dataset. And a griddap request which includes [last] for the time dimension will change if new data arrives for the dataset.
    • Images showing error conditions are cached, but only for a few minutes (it's a difficult situation).
    • Every time a dataset is reloaded, all files in that dataset's cache are deleted. Because requests may be for the "last" index in a gridded dataset, files in the cache may become invalid when a dataset is reloaded.
       
  • Stored Dataset Information -
    For all types of datasets, ERDDAP gathers lots of information when a dataset is loaded and keeps that in memory. This allows ERDDAP to respond very quickly to searches, requests for lists of datasets, and requests for information about a dataset.

    For a few types of datasets (notably EDDGridCopy, EDDTableCopy, EDDGridFromXxxFiles, and EDDTableFromXxxFiles), ERDDAP stores on disk some information about the dataset that is reused when the dataset is reloaded. This greatly speeds the reloading process.

    • Some of the dataset information files are human-readable .json files and are stored in bigParentDirectory/dataset/last2LettersOfDatasetID/datasetID .
    • ERDDAP only deletes these files in unusual situations, e.g., if you add or delete a variable from the dataset's datasets.xml chunk.
    • Most changes to a dataset's datasets.xml chunk (e.g., changing a global attribute or a variable attribute) don't necessitate that you delete these files. A regular dataset reload will handle these types of changes. You can tell ERDDAP to reload a dataset ASAP by setting a flag for the dataset.
    • Similarly, the addition or deletion of files will be handled when ERDDAP reloads a dataset. But ERDDAP will notice this type of change soon and automatically if the dataset is using the <updateEveryNMillis> system.
    • It should only rarely be necessary for you to delete these files. The most common situation where you need to force ERDDAP to delete the stored information (because it is out-of-date/incorrect and won't be automatically fixed by ERDDAP) is when you make changes to the dataset's datasets.xml chunk that affect how ERDDAP interprets data in the source data files, for example, changing the time variable's format string.
    • To delete a dataset's stored information files, run DasDds (which is easier than figuring in which directory the info is located and deleting the files by hand). Then, if ERDDAP is running, set a flag so that the dataset will be reloaded ASAP. Remember that if a dataset is an aggregation of a large number of files, reloading the dataset may take considerable time.
       
  • Unusual Activity: >25% of requests failed -
    You will probably get an email from ERDDAP periodically with the subject "Unusual Activity: >25% of requests failed". It is worth figuring out what is going wrong. The only way to figure out what is going wrong is to read the [bigParentDirectory]/logs/log.txt file and search for "error". For each error message, look just above it to find the line where the original request is logged. The line starts with "{{{{#" and includes the user's IP number and the actual request.

    Use the user's IP number (for example, with http://network-tools.com/ (external link)) to try to figure who or what the user is. Sometimes that will tell you pretty accurately who the user is (e.g., it's a search engine's web crawler). Most of the time it just gives you a clue (e.g., it's an amazonaws computer, it's from some university, it's someone in Washington State (no offense intended; I like Washington)).

    By looking at the actual request, the IP number, and the error message (all from log.txt) for a series of errors, you can usually figure out basically what is going wrong. In my experience, there are three common causes of lots of failed requests:

    1) A search engine is naively trying the URLs listed in ERDDAP web pages and ISO 19115 documents. For example, there are many places which list the base OPeNDAP URL, for example, https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST, to which the user is supposed to add a file type (e.g., .das, .dds, .html). But the search engine doesn't know this. And the request to the base URL fails. A related situation is when the search engine generates bizarre requests or tries to fill out forms in order to get to "hidden" web pages. But the search engines often do a bad job of this, leading to failures. The solution is: create a robots.txt file.

    2) Some user is running a script that is repeatedly asking for something that isn't there. Maybe it is a dataset that used to exist, but is gone now (temporarily or permanently). Scripts often don't expect this and so don't deal with it intelligently. So the script just keeps making requests and the requests keep failing. If you can guess who the user is (from the IP number above), contact them and tell them the dataset is no longer available and ask them to change their script.

    3) Something is really wrong with some dataset. Usually, ERDDAP will make the troubled dataset inactive. Sometimes it doesn't, so all the requests to it just lead to errors. If so, fix the problem with the dataset or (if you can't) set the dataset to active="false". Of course, this may lead to problem #2.

    Sometimes the errors aren't so bad, notably, if ERDDAP can detect the error and respond very quickly (<=1ms). So you may decide to take no action.

    If all else fails, there is a universal solution: add the user's IP number to the <requestBlacklist>. This isn't as bad or as drastic an option as it might seem. The user will then get an error message saying s/he has been blacklisted and telling them your (the ERDDAP administrator's) email address. Sometimes the user will contact you and you can resolve the problem. Sometimes the user doesn't contact you and you will see the exact same behavior coming from a different IP number the next day. Blacklist the new IP number and hope that they will eventually get the message. (Or this is your Groundhog Day, from which you will never escape. Sorry.)

  • robots.txt - The search engine companies use web crawlers (e.g., GoogleBot) to examine all of the pages on the web to add the content to the search engines. For ERDDAP, that is basically good. ERDDAP has lots of links between pages, so the crawlers will find all of the web pages and add them to the search engines. Then, users of the search engines will be able to find datasets on your ERDDAP.

    Unfortunately, some web crawlers (e.g., GoogleBot) are now filling out and submitting forms in order to find additional content. For web commerce sites, this is great. But this is terrible for ERDDAP because it just leads to an infinite number of undesirable and pointless attempts to crawl the actual data. This can lead to more requests for data than from all other users combined. And it fills the search engine with goofy, pointless subsets of the actual data.

    To tell the web crawlers to stop filling out forms and just generally not looking at web pages they don't need to look at, you need to create a text file called robots.txt (external link) in the root directory of your web site's document hierarchy so that it can be viewed by anyone as, e.g., http://www.example.com/robots.txt .
    If you are creating a new robots.txt file, this is a good start:

    User-Agent: *
    Disallow: /erddap/files/ 
    Disallow: /files/ 
    Disallow: /images/ 
    Disallow: /*?
    Disallow: /*?*
    Disallow: /*.asc*
    Disallow: /*.csv*
    Disallow: /*.dods*
    Disallow: /*.esriAscii*
    Disallow: /*.esriCsv*
    Disallow: /*.geoJson*
    Disallow: /*.htmlTable*
    Disallow: /*.json*
    Disallow: /*.mat*
    Disallow: /*.nc*
    Disallow: /*.odvTxt*
    Disallow: /*.tsv*
    Disallow: /*.xhtml*
    Disallow: /*.geotif*
    Disallow: /*.kml*
    Disallow: /*.pdf*
    Disallow: /*.png*
    Disallow: /*.large*
    Disallow: /*.small*
    Disallow: /*.transparentPng*
    Sitemap: http://your.institutions.url/erddap/sitemap.xml
    (But replace your.institutions.url with your ERDDAP's base URL.
    It may take a few days for the search engines to notice and for the changes to take effect.
     
  • sitemap.xml - As the www.sitemaps.org (external link) web site says:
    Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site.

    Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support Sitemaps to pick up all URLs in the Sitemap and learn about those URLs using the associated metadata. Using the Sitemap protocol does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site.

    Actually, since ERDDAP is RESTful, search engine spiders can easily crawl your ERDDAP. But they tend to do it more often (daily!) than necessary (monthly?).
    • Given that each search engine may be crawling your entire ERDDAP every day, this can lead to a lot of unnecessary requests.
    • So ERDDAP generates a sitemap.xml file for your ERDDAP which tells search engines that your ERDDAP only needs to be crawled every month.
    • You should add a refernce to ERDDAP's sitemap.xml to your robots.txt (external link) file:
        Sitemap: http://www.yoursite.org/erddap/sitemap.xml
    • If that doesn't seem to be getting the message to the crawlers, you can tell the various search engines about the sitemap.xml file by visiting these URLs (but change YourInstitution to your institution's acronym or abbreviation and www.yoursite.org to your ERDDAP's URL):
      • http://submissions.ask.com/ping?sitemap=http://www.yoursite.org/erddap/sitemap.xml
      • http://www.bing.com/webmaster/ping.aspx?siteMap=http://www.yoursite.org/erddap/sitemap.xml
      • http://www.google.com/ping?sitemap=http://www.yoursite.org/erddap/sitemap.xml
      • http://search.yahooapis.com/SiteExplorerService/V1/updateNotification?appid=YourInstitution_ERDDAP&url=http://www.yoursite.org/erddap/sitemap.xml
      (I think) you just need to ping each search engine once, for all time. The search engines will then detect changes to sitemap.xml periodically.

     
  • Data Dissemination / Data Distribution Networks: Push and Pull Technology
    • Normally, ERDDAP acts as an intermediary: it takes a request from a user; gets data from a remote data source; reformats the data; and sends it to the user.
    • Pull Technology (external link): ERDDAP also has the ability to actively get all of the available data from a remote data source and store a local copy of the data.
    • Push Technology (external link): By using ERDDAP's subscription services, other data servers can be notified as soon as new data is available so that they can request the data (by pulling the data).
    • ERDDAP's EDDGridFromErddap and EDDTableFromErddap use ERDDAP's subscription services and flag system so that it will be notified immediately when new data is available.
    • You can combine these to great effect: if you wrap an EDDGridCopy around an EDDGridFromErddap dataset (or wrap an EDDTableCopy around an EDDTableFromErddap dataset), ERDDAP will automatically create and maintain a local copy of another ERDDAP's dataset.
    • Because the subscription services work as soon as new data is available, push technology disseminates data very quickly (within seconds).

    This architecture puts each ERDDAP administrator in charge of determining where the data for his/her ERDDAP comes from.

    • Other ERDDAP administrators can do the same. There is no need for coordination between administrators.
    • If many ERDDAP administrators link to each other's ERDDAPs, a data distribution network is formed.
    • Data will be quickly, efficiently, and automatically disseminated from data sources (ERDDAPs and other servers) to data re-distribution sites (ERDDAPs) anywhere in the network.
    • A given ERDDAP can be both a source of data for some datasets and a re-distribution site for other datasets.
    • The resulting network is roughly similar to data distribution networks set up with programs like Unidata's IDD/IDM (external link), but less rigidly structured.
       
  • Security, Authentication, and Authorization - By default, ERDDAP runs as an entirely public server (using http and/or https) with no login (authentication (external link)) system and no restrictions to data access (authorization (external link)).

    If you want to restrict access to some or all datasets to some users, you can use ERDDAP's built-in security system. When the security system is in use:

    • ERDDAP uses role-based access control (external link).
      • The ERDDAP administrator defines users with the <user> tag in datasets.xml. Each user has a username, a password (if authentication=custom), and one or more roles.
      • The ERDDAP administrator defines which roles have access to a given dataset via the <accessibleTo> tag in datasets.xml for any dataset that shouldn't have public access.
    • The user's log in status (and a link to log in/out) will be shown at the top of every web page. (But a logged in user will appear to ERDDAP to be not logged in if he uses an http URL.)
    • If the <baseUrl> that you specify in your setup.xml is an http URL, users who are not logged in may use ERDDAP's http URLs. If <baseHttpsUrl> is also specified, users who are not logged in can also use https URLs.
    • If the <baseUrl> that you specify in your setup.xml is an https URL, users who are not logged are encouraged to use ERDDAP's https URLs -- all of the links on ERDDAP web pages will refer to https URLs. But the only way to force users to use https URLs is by blocking use of the http URLs via Apache + Tomcat. A reason not to force users to use https URLs is: the underlaying SSL/TLS link takes time to establish and then takes time to encrypt and decrypt all information transmitted between the user the the server.
    • Users who are logged in MUST use ERDDAP's https URLs. If they use http URLs, they appear to ERDDAP to be not logged in. This ensures the privacy of the communications and helps prevent session hijacking and sidejacking (external link).
    • Anyone who isn't logged in can access and use the public datasets. By default, private datasets don't appear in lists of datasets if a user isn't logged in. If the administrator has set setup.xml's <listPrivateDatasets> to true, they will appear. Attempts to request data from private datasets (if the user knows the URL) will be redirected to the login page.
    • Anyone who is logged in will be able to see and request data from any public dataset and any private dataset to which their role allows them access. By default, private datasets to which a user doesn't have access don't appear in lists of datasets. If the administrator has set setup.xml's <listPrivateDatasets> to true, they will appear. Attempts to request data from private datasets to which the user doesn't have access will be redirected to the login page.
    • The RSS information for all datasets, even private datasets, is always available to anyone. (This is not ideal. But RSS readers won't ever log in, so they need access without logging in.) But, since private datasets that aren't accessible aren't advertised, their RSS links are not advertised either.
    • Email subscriptions can only be set up when a user has access to a dataset. If a user subscribes to a private dataset, the subscription continues to function after the user has logged out.

    To set up the security/authentication/authorization system:

    • Do the standard ERDDAP initial setup.
    • In setup.xml,
      • Add/change the <authenticate> value from nothing to custom, email, or google (recommended). See the comments about these options below.
      • Add/change the <baseHttpsUrl> value.
      • Insert/uncomment &loginInfo; in <startBodyHtml> to display the user's log in/out info at the top of each web page.
    • For testing purposes on your personal computer, follow these instructions to configure tomcat to support SSL (external link) (the basis for https connections) by creating a keystore with a self-signed certificate (external link) and by modifying tomcat/conf/server.xml to uncomment the connector for port 8443. On Windows, you may need to move .keystore from "c:\Users\you\.keystore" to "c:\Users\Default User\.keystore" or "c:\.keystore" (see tomcat/logs/catalina.today.log if the application doesn't load or users can't see the log in page). You can see when the .keystore certificate will expire by examining the certificate when you log in.

      For a publicly accessible server, instead of using a self-signed certificate, it is strongly recommended that you buy and install a certificate signed by a certificate authority (external link), because it gives your clients more assurance that they are indeed connecting to your ERDDAP, not a man-in-the-middle's version of your ERDDAP. Many vendors sell digital certificates. (Search for web.) They are not expensive.

    • On Linux computers, if Tomcat is running in Apache, modify the /etc/httpd/conf.d/ssl.conf file to allow HTTPS traffic to/from ERDDAP without requiring the :8443 port number in the URL:
      1. Modify the existing <VirtualHost> tag (if there is one), or add one at the end of the file so that it at least has these lines:
        <VirtualHost _default_:443>
           SSLEngine on
           SSLProxyEngine On
           ProxyPass /erddap http://localhost:8443/erddap
           ProxyPassReverse /erddap http://localhost:8443/erddap
        </VirtualHost>
        
      2. Then restart Apache: /usr/sbin/apachectl restart (but sometimes it is in a different directory).
    • In tomcat/conf/server.xml, uncomment the port=8443 <Connector> tag:
      <Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
          maxThreads="150" SSLEnabled="false" scheme="https" secure=“true"
          proxyPort="443" clientAuth="false" sslProtocol="TLS" />
      
    • In datasets.xml, create a <user> tag for each user with username, password (if authorization=custom), and roles information. This is the authorization part of ERDDAP's security system.
    • In datasets.xml, add an <accessibleTo> tag to each dataset that shouldn't have public access. <accessibleTo> lets you specify which roles have access to that dataset.
    • Restart Tomcat. Trouble? Check the Tomcat logs.
    • CHECK YOUR WORK! Any mistake could lead to a security flaw.
    • Check that the login page uses https (not http). Attempts to login via http should be automatically redirected to https and port 8443 (although the port number may be hidden via an Apache proxy). You may need to work with your network administrator to allow external web requests to access port 8443 on your server.
    • You can change the <user> and <accessibleTo> tags at any time. The changes will be applied at the next regular reload of any dataset, or ASAP if you use a flag.

    Authentication (logging in)
    Currently, ERDDAP supports custom, email, and google (recommended) authentication. We strongly recommend the google option because it frees you from storing and handling user's passwords (needed for custom) and is more secure than the email option. Remember that users often use the same password at different sites. So they may be using the same password for your ERDDAP as they do at their bank. That makes their password very valuable -- much more valuable to the user than the data they are requesting. So you need to do as much as you can to keep the passwords private. That is a big responsibility. The email and google options take care of passwords, so you don't have to gather, store, or work with them. So you are freed from that responsibility.

    All authentication options use a cookie (external link) on the user's computer, so the user's browser must be set to allow cookies. If a user is making ERDDAP requests from a computer program (not a browser), cookies are hard to work with. Sorry.

    The details of the authentication options are:

    • custom
      custom is ERDDAP's custom system for letting users log in by entering their User Name and Password in a form on a web page. If a user tries and fails to log in 3 times within 10 minutes, the user is blocked from trying to log in for 10 minutes. This prevents hackers from simply trying millions of passwords until they find the right one.

      This is secure because the User Name and Password are transmitted via https (not http), but authentication=google is better because it frees you from having to handle passwords. The custom approach requires you to collect a user's Name and a hash digest of their Password (use your phone! email isn't secure!) and store them in datasets.xml in <user> tags.

      With the custom option, no one can log in until you (the ERDDAP administrator) create a <user> tag for the user, specifying the user's name as the username, the hash digest of their password as the password, and their roles.

      Not Recommended
      Because of the awkwardness of generating and transmitting the hash digest of the user's password and because of the risks associated with ERDDAP holding the hash digests of the passwords, this option is not recommended.

      To increase the security of this option:

      • Use UEPSHA256 for <passwordEncoding> in setup.xml.
      • Use an as-secure-as-possible method to pass the hash digest of the user's password from the user to the ERDDAP administrator (phone?).
      • Make sure that other users on the server (i.e., Linux users, not ERDDAP users) can't read files in the Tomcat directory or ERDDAP's bigParentDirectory.
        On Linux, as user=tomcat, use:
        chmod -R g-rwx bigParentDirectory
        chmod -R g-rwx tomcatDirectory
         
    • email
      The email authentication option uses a user's email account to authenticate the user (by sending them an email with a special link that they have to access in order to log in). Unlike other emails that ERDDAP sends, ERDDAP does not write these invitation emails to the email log file because they contain confidential information.
      In theory, this is not very secure, because emails aren't always encrypted, so a bad guy with the ability to intercept emails could abuse this system by using a valid user's email address and intercepting the invitation email.
      In practice, if you set up ERDDAP to use a Google email account to send emails, and if you set it up to use one of the TLS options for the connection, and if the user has a Google email account, this is somewhat secure because the emails are encrypted all the way from ERDDAP to the user.

      To increase the security of this option:

      • Make sure that other users on the server (i.e., Linux users, not ERDDAP users) can't read files in the Tomcat directory or ERDDAP's bigParentDirectory.
        On Linux, as user=tomcat, use:
        chmod -R g-rwx bigParentDirectory
        chmod -R g-rwx tomcatDirectory
      • Set things up to get end-to-end security for the emails sent from ERDDAP to the users. For example, you could make a Google-centric system by only creating <user> tags for Google-managed email addresses and by setting up your ERDDAP to use a Google email server via a secure/TLS connection: in your setup.xml, use e.g.,
        <emailSmtpHost>smtp.gmail.com</emailSmtpHost>
        <emailSmtpPort>587</emailSmtpPort>
        <emailProperties>mail.smtp.starttls.enable|true</emailProperties>

      Not Recommended
      We only recommend the email authentication option only as a temporary alternative to the google authentication option, and only if you follow the security suggesions above. The weakness of the google option is that it relies on an external authentication system (Google Sign-In), which may suddenly cease to work with ERDDAP, leaving your users unable to log in. In that case, switching temporarily to the email option solves the problem until the google option is again working.

      As with the google option, email is very convenient for ERDDAP administrators -- you don't ever have to deal with passwords or their hash digests. All you need to create is a <user> tag for a user in datasets.xml is the user's Google email address, which ERDDAP uses as the user's name. (The password attribute isn't used when authentication=email or google.)

      With the email option, only users that have a <user> tag in datasets.xml can try to log in to ERDDAP by providing their email address and clicking on the link in the email that ERDDAP sends them.

      ERDDAP treats email addresses as case-insensitive. It does this by converting email addresses you enter (in <user> tags) or users enter (on the login form) to their all lowercase version.

      To set up authentication=email:

      • In your setup.xml, change the <baseHttpsUrl> tag's value.
        For experimenting/working on your personal computer, use
        https://localhost:8443
        For your public ERDDAP, use
        https://your.domain.org:8443
        or without the :8443 if you are using an Apache proxypass so that the port number isn't needed.
      • In your setup.xml, change the <authentication> tag's value to email:
        <authentication>email</authentication>
      • In your setup.xml, make sure the email system is setup via all of the <email...> tags, so that ERDDAP can send out emails. If possible, set this up to use a secure connection (SSL / TLS) to the email server.
      • In your datasets.xml, create <user> tags for each user who will have access to private datasets.
        Use the user's email address as the username in the tag.
        Don't specify the password attribute for the user.
      • Restart ERDDAP so that the changes to setup.xml and datasets.xml take effect.
         
    • google (recommended)
      The google option uses Google Sign-In (external link), which is based on OAuth 2.0 (external link), the current version of the OAuth protocol.

      The google option lets users log in by signing into their Google email account, including Google-managed accounts such as @noaa.gov accounts. This allows ERDDAP to verify the user's identity (name and email address) and access their profile image, but does not give ERDDAP access to their emails, their Google Drive, or any other private information.

      The google option is the successor to the openid option, which was discontinued after ERDDAP version 1.68, and which was based on a version of openID that is now out-of-date. Please switch to the google option.

      Google Sign-In is very convenient for ERDDAP administrators -- you don't ever have to deal with passwords or their hash digests. All you need to create is a <user> tag for a user in datasets.xml which specifies the user's Google email address as the username attribute. (The password attribute isn't used when authentication=google or email.)

      With the google option, anyone can log in to ERDDAP by signing into their Google email account, but no one will have the right to access private datasets until you (the ERDDAP administrator) create a <user> tag, specifying their email address as the username and specifying their roles.

      Recommended
      This is the authentication option which we believe is by far the most secure option. The other options have significantly weaker security.

      ERDDAP treats email addresses as case-insensitive. It does this by converting email addresses you enter (in <user> tags) or users enter (on the login form) to their all lowercase version.

      To set up authentication=google:

      • In your setup.xml, change the <baseHttpsUrl> tag's value.
        For experimenting/working on your personal computer, use
        https://localhost:8443
        For your public ERDDAP, use
        https://your.domain.org:8443
        or without the :8443 if you are using an Apache proxypass so that the port number isn't needed.
      • In your setup.xml, change the <authentication> tag's value to google:
        <authentication>google</authentication>
      • If you don't have a Google email account, create one (external link)
      • Follow these instructions (external link) to create a Google Developers Console project and get a client ID.

        For authorized JavaScript origins, enter the value from <baseHttpsUrl> from your personal computer's ERDDAP setup.xml, e.g.,
        https://localhost:8443
        On a second line, add the <baseHttpsUrl> from your public ERDDAP setup.xml, e.g.,
        https://your.domain.org:8443

        Don't specify any Authorized redirect URIs.

        When you see your Client ID for this project, copy and paste it into your setup.xml (usually just below <authentication> to be orderly, but placement doesn't actually matter), in the <googleClientID> tag, e.g.,
        <googleClientID>yourClientID</googleClientID>
        The client ID will be a string of about 75 characters, probably starting with several digits and ending with .apps.googleusercontent.com .

      • In your datasets.xml, create <user> tags for each user who will have access to private datasets.
        Use the user's email address as the username in the tag.
        Don't specify the password attribute for the user.
      • Restart ERDDAP so that the changes to setup.xml and datasets.xml take effect.

      The google option relies on Google Sign-In. If that suddenly ceases to work (for whatever reason), users won't be able to sign in to your ERDDAP. As a temporary solution, you can (it's up to you) switch to ERDDAP's email authentication option temporarily. The <user> tag setup is the same, so the only thing you need to do is change the <authentication> tag's value in setup.xml, to
      <authentication>email</authentication>

    ERDDAP no longer supports the openid authentication option, which was based on a version of openID that is now out-of-date. Please use the google option instead.

    ERDDAP doesn't support BASIC authentication because:

    • BASIC seems geared toward predefined web pages needing secure access or blanket on/off access to the whole site, but ERDDAP allows (restricted access) datasets to be added on-the-fly.
    • BASIC authentication doesn't offer a way for users to log out!
    • BASIC authentication is known to be not secure.

    Secure Data Sources
    If a data set is to have restricted access to ERDDAP users, the data source (from where ERDDAP gets the data) should not be publicly accessible. So how can ERDDAP get the data for restricted access datasets? Some options are:

    • ERDDAP can serve data from local files (for example, via EDDTableFromFiles or EDDGridFromFiles).
       
    • ERDDAP can be in a DMZ (external link) and the data source (e.g., an OPeNDAP server or a database) can be behind a firewall (external link), where it is accessible to ERDDAP but not to the public.
       
    • The data source can be on a public web site, but require a login to get the data. The two types of dataset that ERDDAP can log on to access is EDDTableFromDatabase and EDDTableFromCassandra. These datasets support (and should always use) user names (create an ERDDAP user who only has read-only privileges), passwords, SSL connections, and other security measures.

      But in general, currently, ERDDAP can't deal these data sources because it has no provisions for logging on to the data source. This is the reason why access to EDDGridFromErddap and EDDTableFromErddap datasets can't be restricted. Currently, the local ERDDAP has no way to login and access the metadata information from the remote ERDDAP. And putting the "remote" ERDDAP behind your firewall and removing that dataset's accessibleTo restrictions doesn't solve the problem: since user requests for EDDXxxFromErddap data need to be redirected to the remote ERDDAP, the remote ERDDAP must be accessible.

    Defenses Against Hackers
    There are bad guy hackers who try to exploit security weaknesses in server software like ERDDAP. ERDDAP follows the common security advice to have several layers of defenses:

    • Restricted Privileges - One of the most important defenses is to run Tomcat via a user called tomcat that doesn't have a password (so no one can log in as that user) and has limited file system privileges (e.g., read-only access to the data). See ERDDAP's instructions for setting up tomcat.
    • Heavy Use - In general, ERDDAP is built for heavy use, including by scripts which make tens of thousands of requests, one after another. It is hard for ERDDAP to simultaneously open itself up to heavy legitimate use and shield itself from abuse. It is sometimes hard to differentiate heavy legitimate use, excessive legitimate use, and illegitimate use (and sometimes it is really easy). Among other defenses, ERDDAP consciously does not allow a single request to use an inordinate fraction of the system's resources (unless the system is otherwise not active).
    • Identify Troublesome Users - If ERDDAP is slowing down or freezing (perhaps because a naive user or a bot is running multiple scripts to submit multiple requests simultaneously or perhaps because of a bad guy's Denial-of-service attack), you can look at the Daily Report email (and more requent identical information in the ERDDAP log file) which displays the number of requests made by the most active users (see "Requester's IP Address (Allowed)"). ERDDAP also sends emails to the administrator whenever there is "Unusual activity: >25% of requests failed". You can then look in the ERDDAP log file to see the nature of their requests. If you feel that someone is making too many requests, bizarre requests (you wouldn't believe what I've seen, well, maybe you would), or attack-type requests, you can add their IP address to the blacklist.
    • Blacklist - You can add the IP address of troublesome users, bots, and Denial-of-service attackers to the ERDDAP blacklist, so that future requests from them will be immediately rejected. This setting is in datasets.xml so that you can quickly add an IP address to the list and then flag a dataset so that ERDDAP immediately notices and applies the change. The error message sent to blacklisted users encourages them to contact the ERDDAP administrator if they feel they have been mistakenly put on the blacklist. (In our experience, several users have been unaware that they were running multiple scripts simultaneously, or that their scripts were making nonsense requests.)
    • Dataset Security - Some types of datasets (notably, EDDTableFromDatabase) present additional security risks (e.g., SQL injection) and have their own security measures. See the information for those types of datasets in Working with the datasets.xml File, notably EDDTableFromDatabase security.
    • Security Audit - I (Bob) tried repeatedly for 3+ years to get NOAA IT security to do a security audit of ERDDAP. They have repeatedly refused. At one point, they said they only do that for software developed in the DC area. Fortunately, in 2012, a group in USGS started to use ERDDAP. In early 2013, they hired the security firm Acunetix to do a security scan of ERDDAP. They found some problems. I fixed some of them and gave reasons for why the other issues were false alerts. They redid the scan. If you would like to see the report and my reply, please email bob dot simons at noaa dot gov.

    Questions? Suggestions?
    If you have any questions about ERDDAP's security system or have any questions, doubts, concerns, or suggestions about how it is set up, please email bob dot simons at noaa dot gov.

Things You Don't Need To Know

These are details that you don't need to know until a need arises.
  • Setting Up a Second ERDDAP for Testing/Development
    If you want to do this, there are two approaches:
    • (Best) Install Tomcat and ERDDAP on a computer other than the computer that has your public ERDDAP. If you use your personal computer:
      1. Do the installation one step at a time. Get Tomcat up and running first.
        When Tomcat is running, the Tomcat Manager should be at
        http://127.0.0.1:8080/manager/html/
      2. Install ERDDAP.
      3. Don't use ProxyPass to eliminate the port number from the ERDDAP URL.
      4. In setup.xml, set baseUrl to http://127.0.0.1:8080
      5. After you start up this ERDDAP, you should be able to see it at
        http://127.0.0.1:8080/erddap
    • (Second Best) Install another Tomcat on the same computer as your public ERDDAP.
      1. Do the installation one step at a time. Get Tomcat up and running first.
        Change all of the port numbers associated with the second Tomcat (e.g., change 8080 to 8081) (see these directions (external link)).
      2. Install ERDDAP in the new Tomcat.
      3. Don't use ProxyPass to eliminate the port number from the ERDDAP URL.
      4. In setup.xml, set baseUrl to http://www.yourDomainName:8081
      5. After you start up this ERDDAP, you should be able to see it at
        http://www.yourDomainName:8081/erddap
         
  • Heavy Loads / Constraints - With heavy use, a standalone ERDDAP may be constrained by various problems. For more information, see the list of constraints and solutions.
     
  • Grids, Clusters, and Federations - Under very heavy use, a single standalone ERDDAP will run into one or more constraints and even the suggested solutions will be insufficient. For such situations, ERDDAP has features that make it easy to construct scalable grids (also called clusters or federations) of ERDDAPs which allow the system to handle very heavy use (e.g., for a large data center). For more information, see grids, clusters, and federations of ERDDAPs.
     
  • Cloud Computing - Several companies are starting to offer cloud computing services (external link) (e.g., Amazon Web Services (external link)). Web hosting companies (external link) have offered simpler services since the mid-1990's, but the "cloud" services have greatly expanded the flexibility of the systems and the range of services offered. You can use these services to set up a single ERDDAP or a grid/cluster of ERDDAPs to handle very heavy use. For more information, see cloud computing with ERDDAP.
     
  • Amazon Web Services (AWS) EC2 Installation Overview - Amazon Web Services (AWS) (external link) is a cloud computing service (external link) that offers a wide range of computer infrastructure that you can rent by the hour. You can install ERDDAP on an Elastic Compute Cloud (EC2) (external link) instance (their name for a computer that you can rent by the hour). AWS has an excellent AWS User Guide (external link) and you can use Google to find answers to specific questions you might have. Brace yourself -- it is a fair amount of work to get started. But once you get one server up and running, you can easily rent as many additional resources (servers, databases, SSD-space, etc.) as you need, at a reasonable price. [This isn't a recommendation or endorsement of Amazon Web Services. There are other cloud providers.]

    An overview of things you need to do to get ERDDAP running on AWS is:

    • In general, you will do all the things described in the AWS User Guide (external link).
    • Set up an AWS account.
    • Set up an AWS user within that account with administrator privileges. Log in as this user to do all the following steps.
    • Elastic Block Storage (EBS) is the AWS equivalent of a hard drive attached to your server. Some EBS space will be allocated when you first create an EC2 instance. It is persistent storage -- the information isn't lost when you stop your EC2 instance. And if you change instance types, your EBS space automatically gets attached to the new instance.
    • Create an Elastic IP address so that your EC2 instance has a stable, public URL (as opposed to just a private URL that changes every time you restart your instance).
    • Create and start up an EC2 instance (computer). There are a wide range of instance types (external link), each at a different price. An m4.large or m4.xlarge instance is powerful and is probably suitable for most uses, but choose whatever meets your needs. You will probably want to use Amazon's Linux as the operating system.
    • If your desktop/laptop computer is a Windows computer, you can use PuTTY (external link), a free SSH client for Windows, to get access to your EC2 instance's command line. Or, you may have some other SSH program that you prefer.
    • When you log into your EC2 instance, you will be logged in as the administrative user with the user name "ec2-user". ec2-user has sudo privileges. So, when you need to do something as the root user, use: sudo someCommand
    • If your desktop/laptop computer is a Windows computer, you can use FileZilla (external link), a free SFTP program, to transfer files to/from your EC2 instance. Or, you may have some other SFTP program that you prefer.
    • Install Apache (external link) on your EC2 instance.
    • Follow the standard ERDDAP installation instructions.
       
  • WaitThenTryAgain Exception - In unusual circumstances, a user may get an error message like
      WaitThenTryAgainException:
      There was a (temporary?) problem. Wait a minute, then try again. (In a browser, click the Reload button.)
      Details: GridDataAccessor.increment: partialResults[0]="123542730" was expected to be "123532800".

    The explanation is: For each EDDGrid dataset, ERDDAP keeps the axis variable values in memory. They are used, for example, to convert requested axis values that use the "()" format into index numbers. For example, if the axis values are "10, 15, 20, 25", a request for (20) will be interpreted as a request for index #2 (0-based indices). When ERDDAP gets a request for data and gets the data from the source, it verifies that the axis values that it got from the source match the axis values in memory. Normally, they do. But sometimes the data source has changed in a significant way: for example, index values from the beginning of the axis variable may have been removed (e.g., "10, 15, 20, 25" may have become "20, 25, 30"). If that happens, it is clear that ERDDAP's interpretation of the request (e.g., "(20)" is index #2) is now wrong. So ERDDAP throws an exception and marks the dataset as needing to be reloaded. ERDDAP will update the dataset soon (often in a few seconds, usually within a minute). Other, similar problems also throw the WaitThenTryAgain exception.

    Starting with ERDDAP version 1.14, it became much less likely that a user would actually see this error. Now, when the underlying error occurs, ERDDAP automatically internally tries to reload the dataset and resubmit the request to the reloaded dataset. Often this succeeds. When it does, the user will simply see that a given request took a little longer than usual. If it fails, the user should (as the message says) wait a minute, then try again.

  • Palettes are used to convert a range of numbers into a range of colors when making graphs and maps. ERDDAP comes with several pre-made palettes. You can add additional palettes. ERDDAP's palettes are defined in GMT (external link)-style .cpt (Color Palette Table) files (external link). All ERDDAP .cpt files are valid GMT .cpt files, but the opposite may not be true. ERDDAP may just support a subset of what GMT supports. If there is trouble, ERDDAP will probably throw an error when the .cpt file is parsed (which is better than misusing the information). ERDDAP requires that all .cpt files be stored in tomcat/webapps/erddap/WEB-INF/cptfiles. To get ERDDAP to use a new .cpt file, store the file in that directory and either:
    • If you use the default messages.xml file: add the file name to the <palettes> tag in
      tomcat/webapps/erddap/WEB-INF/classes/gov/noaa/pfel/erddap/util/messages.xml.
      If you do this, you need to do it every time you upgrade ERDDAP.
    • If you use a custom messages.xml file: add the file name to the <palettes> tag in your custom messages.xml file: tomcat/content/erddap/messages.xml . If you do this, you only need to do it once (but there is other work to maintain a custom messages.xml file).

    But don't remove any of the standard palettes. They are a standard feature of all ERDDAP installations. An advantage of this approach is that you can specify the order of the palettes in the list presented to users.
    And for every new ERDDAP release, you need to put your .cpt files in tomcat/webapps/erddap/WEB-INF/cptfiles and restart ERDDAP so ERDDAP findes the files.
     
  • How does ERDDAP generate the colors in a colorbar?
    1. The user selects one of the predefined palettes or uses the default, e.g., Rainbow. Palettes are stored/defined in GMT-style .cpt Color Palette Table files. Each of ERDDAP's predefined palettes has a simple integer range, e.g., 0 to 1 (if there is just one section in the palette), or 0 to 4 (if there are four sections in the palette). Each segment in the file covers n to n+1, starting at n=0.
    2. ERDDAP generates a new .cpt file on-the-fly, by scaling the predefined palette's range (e.g., 0 to 4) to the range of the palette needed by the user (e.g., 0.1 to 50) and then generating a section in the new palette for each section of the new palette (e.g., a log scale with ticks at 0.1, 0.5, 1, 5, 10, 50 will have 5 sections). The color for the end point of each section is generated by finding the relevant section of the palette in the .cpt file, then linearly interpolating the R, G, and B values. (That's the same as how GMT generates colors from its Color Palette Table files.) This system allows ERDDAP to start with generic palettes (e.g., Rainbow with 8 segments, in total spanning 0 to 8) and create custom palettes on-the-fly (e.g., a custom Rainbow, which maps 0.1 to 50 mg/L to the rainbow colors).
    3. ERDDAP then uses that new .cpt file to generate the color for each different colored pixel in the color bar (and later for each data point when plotting data on a graph or map), again by finding the relevant section of the palette in the .cpt file, then linearly interpolating the R, G, and B values.
    This process may seem unnecessarily complicated. But it solves problems related to log scales that are hard to solve other ways.

    So how can you mimic what ERDDAP is doing? That isn't easy. Basically you need to duplicate the process that ERDDAP is using. If you are a Java programmer, you can use the same Java class that ERDDAP uses to do all of this:
    tomcat/webapps/erddap/WEB-INF/classes/gov/noaa/pfel/coastwatch/sgt/CompoundColorMap.java.

  • Guidelines for Data Distribution Systems - More general opinions about the design and evaluation of data distribution systems can be found here.
     
  • ArchiveADataset - Included in your ERDDAP installation is a command line tool called ArchiveADataset which can help you make an archive (a .tar.gz file) with part or all of a dataset stored in a series of data files in a file format that is suitable for submission to NOAA's NCEI archive (.nc for gridded datasets or .ncCFMA for tabular datasets, as specified by the NCEI NetCDF Templates v2.0 (external link)).

    ArchiveADataset follows these NCEI Archiving Guidelines (external link), this guide for Archiving Your Data at NCEI (external link), and the related Practices for Ensuring Data Integrity (external link). Not surprisingly, the global and variable metadata that ERDDAP encourages/requires is almost exactly the same metadata that NCEI encourages/requires, so all of your datasets should be ready for submission to NCEI via Send2NCEI (external link) or ATRAC (external link) (NCEI's Advanced Tracking and Resource tool for Archive Collections). Also, NCEI can download the ISO 19115 .xml metadata file for the dataset from your ERDDAP.

    ArchiveADataset may be useful in other situations, for example, for ERDDAP administrators who need to convert a subset of a dataset (on a private ERDDAP) from its native file format into a set of .ncCF files, so that a public ERDDAP can serve the data from the .ncCF files instead of the original files.

    Once you have set up ERDDAP and run it (at least one time), you can find and use ArchiveADataset in the tomcat/webapps/erddap/WEB-INF directory. There is a shell script (ArchiveADataset.sh) for Linux/Unix and a batch file (ArchiveADataset.bat) for Windows. When you run ArchiveADataset, it will ask you a series of questions. For each question, type a response, then press Enter. Or press ^C to exit a program at any time.

    Or, you can put the answers to the questions, in order, on the command line. To do this, run the program once and type in and write down your answers. Then, you can create a single command line (with the answers as parameters) which runs the program and answers all the questions.
    Use the word default if you want to use the default value for a given parameter.
    Use "" (two double quotes) as a placeholder for an empty string.
    Specifying parameters on the command line can be very convenient, for example, if you use ArchiveADataset once a month to archive a month's worth of data. Once you have generated the command line with parameters and saved that in your notes or in a shell script, you just need to make small changes each month to make that month's archive.

    The questions that ArchiveADataset asks allow you to:

    • Specify a contact email address for this archive (it will be written in the READ_ME.txt file in the archive).
    • Specify the datasetID of the dataset you want to archive.
    • Specify which data variables you want to archive (usually all).
    • Specify which subset of the dataset you want to archive. You need to format the subset in the same way you would format a subset for a data request, so it will be different for gridded than for tabular datasets.
      • For gridded datasets, you can specify a range of values of the leftmost dimension, usually that is a range of time. ArchiveADataset will make a separate request and generate a separate data file for each value in the range of values. Since gridded datasets are usually large, you will almost always have to specify a small subset relative to the size of the entire dataset.
        For example, [(2015-12-01):(2015-12-31)][][][]
      • For tabular datasets, you can specify any collection of constraints, but it is often a range of time. Since tabular datasets are usually small, it is often possible to specify no constraints, so that the entire dataset is archived.
        For example, &time>=2015-12-01&time<2016-01-01
    • For tabular datasets: specify a comma separated list of 0 or more variables that will determine how the archived data is further subsetted into different data files. For datasets that have
      cdm_data_type=TimeSeries|TimeSeriesProfile|Trajectory|TrajectoryProfile
      you should almost always specify the variable that has the cf_role=timeseries_id (e.g., stationID) or cf_role=trajectory_id attibute. ArchiveADataset will make a separate request and generate a separate data file for each combination of the values of these variables, e.g., for each stationID.
      For all other tabular datasets, you will probably not specify any variables for this purpose.
      Warning: If the subset of the dataset you are archiving is very large (>2GB) and there is no suitable variable for this purpose, then ArchiveADataset is not usable with this dataset. This should be rare.
    • Specify the file format for the data files that will be created.
      For gridded datasets, for NCEI, use .nc .
      For tabular datasets, for NCEI, use .ncCFMA.
    • Specify the type of file digest to be created for each data file and for the .tar.gz file. The file digest provides a way for the client (e.g., NCEI) to test whether the data file has become corrupted. Traditionally, these were .md5 files (external link), but now there are better options. For NCEI, use .sha256 .

    After you answer all of the questions, ArchiveADataset will:

    1. Make a series of requests to the dataset and stage the resulting data files in bigParentDirectory/ArchiveADataset/datasetID_timestamp/.
      For gridded datasets, there will be a file for each value of the leftmost dimension (e.g., time). The name of the file will be that value (e.g., the time value).
      For tabular datasets, there will be a file for each value of the ... variable(s). The name of the file will be that value. If there is more than one variable, the left variables will be used to make subdirectory names, and the rightmost variable will be used to make the file names.
      Each of data file must be <2GB (the maximum allowed by .nc version 3 files).
    2. Make a file related to each data file with the digest of the data file. For example, if the data file is 46088.nc and the digest type is .sha256, then the digest file will have the name 46088.nc.sha256 .
    3. Make a READ_ME.txt file with information about the archive, including a list of all the settings you specified to generate this archive.
    4. Make 3 files in bigParentDirectory/ArchiveADataset/ :
      • A .tar.gz archive file named datasetID_timestamp.tar.gz containing all of the staged data files and digest files. The .tar.gz file may be any size, limited only by disk space.
      • A digest file for the .tar.gz file, for example, datasetID_timestamp.tar.gz.sha256
      • A text file named datasetID_timestamp.tar.gz.listOfFiles.txt which lists all of the files in the .tar.gz file.
      If you are preparing the archive for NCEI, these are the files that you will send to NCEI, perhaps via Send2NCEI (external link) or ATRAC (external link) (NCEI's Advanced Tracking and Resource tool for Archive Collections).
    5. Delete all of the staged files so that only the .tar.gz, .tar.gz.sha256, and .tar.gz.listOfFiles.txt files remain.

    Problems? Suggestions? ArchiveADataset is new. If you have problems or suggestions, please email them to bob dot simons at noaa dot gov .
     

Slide Shows

Here are some PowerPoint slide shows and documents that Bob Simons has created related to ERDDAP.

DISCLAIMER: The content and opinions expressed in these documents are Bob Simons' personal opinions and do not necessarily reflect any position of the Government or the National Oceanic and Atmospheric Administration.

The Three Main Documents:

Previous/Specialized Presentations:

I hope it isn't true that "Every time you make a PowerPoint, Edward Tufte kills a kitten." (external link)
 

Programmer's Guide

These are things that only a programmer who intends to work with ERDDAP's Java classes needs to know.
  • Source Code
    • The source code for the current version of ERDDAP is always in the current erddap.war file. So when you install ERDDAP in tomcat, all of the source code is unpacked and installed on the server. If you just want to read the source code, this is the easiest option.
    • The source code for recent public versions and in-development versions is also available via GitHub (external link). Please read the Wiki (external link) for that project. If you want to modify the source code (and possibly have the changes incorporated into the standard ERDDAP distribution), this is the recommended approach.
    • ERDDAP and its subcomponents have very liberal, open-source licenses, so you can use and modify the source code for any purpose, for-profit or not-for-profit. Note that ERDDAP and many subcomponents have licenses that require that you acknowledge the source of the code that you are using. See Credits. Whether required or not, it is just good form to acknowledge all of these contributors.
       
  • Use the Code for Other Projects While you are welcome to use parts of the ERDDAP code for other projects, be warned that the code can and will change. We don't promise to support other uses of our code. Git and GitHub will be your main solutions for dealing with this - Git allows you to merge our changes into your changes.

    For many situations where you might be tempted to use parts of ERDDAP in your project, we think you will find it much easier to install and use ERDDAP as is, and then write other services which use ERDDAP's services. You can set up your own ERDDAP installation crudely in an hour or two. You can set up your own ERDDAP installation in a polished way in a few days (depending on the number and complexity of your datasets). But hacking out parts of ERDDAP for your own project is likely to take weeks (and months to catch subtleties) and you will lose the ability to incorporate changes and bug fixes from subsequent ERDDAP releases. We (obviously) think there are many benefits to using ERDDAP as is and making your ERDDAP installation publicly accessible. However, in some circumstances, you might not want to make your ERDDAP installation publicly accessible. Then, your service can access and use your private ERDDAP and your clients needn't know about ERDDAP.

    Half Way - Or, there is another approach which you may find useful which is half way between delving into ERDDAP's code and using ERDDAP as a stand-alone web service: In the EDD class, there is a static method which lets you make an instance of a dataset (based on the specification in datasets.xml):
      oneFromDatasetXml(String tDatasetID)
    It returns an instance of an EDDTable or EDDGrid dataset. Given that instance, you can call
      makeNewFileForDapQuery(String userDapQuery, String dir, String fileName, String fileTypeName)
    to tell the instance to make a data file, of a specific fileType, with the results from a user query. Thus, this is a simple way to use ERDDAP's methods to request data and get a file in response, just as a client would use the ERDDAP web application. But this approach works within your Java program and bypasses the need for an application server like Tomcat. We use this approach for many of the unit tests of EDDTable and EDDGrid subclasses, so you can see examples of this in the source code for all of those classes.

  • Development Environment
    • Directory Structure
      • If you are installing ERDDAP in a Tomcat (whether or not you will actually use it that way), follow the instructions above.
      • If you aren't installing ERDDAP in a Tomcat, you still need to make a Tomcat-like directory structure, so that ERDDAP can find the setup files in tomcat/content/erddap .
        • Make a directory somewhere called "tomcat" (it can be something else, but this is easier to explain).
        • As indicated above, unzip erddapContent.zip into tomcat, creating tomcat/content/erddap . Follow the instructions above to modify setup.xml and datasets.xml. Depending on your situation, you may need to specify that directory by adding something like
            -DErddapContentDirectory="/someDirectory/content/erddap/"
          to the Java command line for your program.
        • Make a tomcat/webapps/erddap directory.
        • .war files are just .zip files that follow a few additional conventions. So you can use an unzip program to unzip erddap.war into tomcat/webapps/erddap . That has all of ERDDAP's .java classes and many other files.
    • Our development environment is just a programmer's editor (we're not saying which one). (No, we don't use Eclipse, Ant, Maven, or ...; nor do we offer ERDDAP-related support for them. If we required you to use Ant and you preferred Maven (or vice versa), you wouldn't be very happy about it.)
    • We use a batch file which deletes all of the .class files in the source tree.
    • We currently use javac 1.8.0_77 to compile gov.noaa.pfel.coastwatch.TestAll (it has links to a few classes that wouldn't be compiled otherwise) and java 1.8.0_77 to run the tests. For security reasons, it is almost always best to use the latest versions of Java and Tomcat.
      • When we run javac or java, the current directory is tomcat/webapps/erddap/WEB-INF .
      • Our javac and java classpath (including some unnecessary items) is currently
        ./classes;../../../lib/servlet-api.jar;lib/axis.jar;lib/cassandra-driver-core.jar;lib/netty-all.jar;lib/guava.jar;lib/metrics-core.jar;lib/lz4.jar;lib/snappy-java.jar;lib/commons-compress.jar;lib/commons-discovery.jar;lib/itext-1.3.1.jar;lib/jaxrpc.jar;lib/lucene-core.jar;lib/mail.jar;lib/netcdfAll-latest.jar;lib/slf4j.jar;lib/postgresql.jdbc.jar;lib/saaj.jar;lib/wsdl4j.jar;lib/aws-java-sdk.jar;lib/commons-codec.jar;lib/commons-logging.jar;lib/fluent-hc.jar;lib/httpclient.jar;lib/httpclient-cache.jar;lib/httpcore.jar;lib/httpmime.jar;lib/jna.jar;lib/jna-platform.jar;lib/jackson-annotations.jar;lib/jackson-core.jar;lib/jackson-databind.jar
        If you get error messages about classes not found when you compile or run ERDDAP or one of its tools, compare your command line's classpath to the classpath above to figure out which .jars are missing from your classpath.
      • So your javac command line will be something like
        javac -cp (classpath above) classes/gov/noaa/pfel/coastwatch/TestAll.java
      • And your java command line will be something like
        java -cp (classpath above) -Xmx1200M -Xms1200M classes/gov/noaa/pfel/coastwatch/TestAll
        Optional: you can add -verbose:gc. It tells Java to print garbage collection statistics.
      • If TestAll compiles, everything ERDDAP needs has been compiled. Lots of classes are compiled that aren't needed for ERDDAP. If compiling TestAll succeeds but doesn't compile some class, that class isn't needed. (There are some unfinished/unused classes.)
    • We use some 3rd party source code instead of .jar files (notably for Dods) and have modified them slightly to avoid problems compiling with Java 1.8. We have often made other slight modifications (notably to Dods) for other reasons.
    • Most classes have test methods. We run lots of tests via TestAll. Unfortunately, many of the tests are specific to our set up. (Sorry.)
       
  • Important Classes - If you want to look at the source code and try to figure out how ERDDAP works, please do.
    • The code has JavaDoc comments, but the JavaDocs haven't been generated. Feel free to generate them.
    • The most important classes (including the ones mentioned below) are within gov/noaa/pfel/erddap.
    • The Erddap class has the highest level methods. It extends HttpServlet.
    • Erddap passes requests to instances of subclasses of EDDGrid or EDDTable, which represent individual datasets.
    • EDDGrid and EDDTable subclasses parse the request, get data from subclass-specific methods, then format the data for the response.
    • EDDGrid subclasses push data into GridDataAccessor (the internal data container for gridded data).
    • EDDTable subclasses push data into TableWriter subclasses, which write data to a specific file type on-the-fly.
       
  • Code Contributions - If you have written some code which would be a nice addition to ERDDAP, please email bob dot simons at noaa dot gov. We'll work out the details. The two situations that are most likely are:
    1. You want to write another subclass of EDDGrid or EDDTable to handle another data source type. If so, we recommend that you find the closest existing subclass and use that code as a starting point.
    2. You want to write another saveAsFileType method. If so, we recommend that you find the closest existing saveAsFileType method in EDDGrid or EDDTable and use that code as a starting point.
    Both of these situations have the advantage that the code you write is self-contained. You won't need to know all the details of ERDDAP's internals. And it will be easy for us to incorporate your code in ERDDAP. Note that if you do submit code, the license will need compatible with the ERDDAP license (e.g., Apache (external link), BSD (external link), or MIT-X (external link)). You can hold the copyright to your code. We'll list your contribution in the credits.
     

List of Changes

The List of Changes for each ERDDAP release is now on a separate web page.

Credits

ERDDAP is a product of the NOAA (external link) NMFS (external link) ERD.
Bob Simons is the author of ERDDAP (the designer and programmer who wrote the ERDDAP-specific code).
Roy Mendelssohn instigated the project and provides essential ongoing support (including hardware, network, and other software support) that has made this project possible and facilitated this probject, including by freeing up Bob's time so he could spend more time on the ERDDAP code.
The ERDDAP-specific code is licensed as copyrighted open source, with NOAA (external link) holding the copyright. See the ERDDAP license.
ERDDAP uses copyrighted open source, Apache, LGPL, MIT/X, Mozilla, and public domain libraries and data.
ERDDAP does not require any GPL code or commercial programs.

The bulk of the funding for work on ERDDAP has come from NOAA, in that it pays Bob Simons salary (he is a full time government employee with "hard"-money/full funding). In the early days of ERDDAP, when he was a government contractor, funding came from the NOAA CoastWatch (external link) program, the NOAA IOOS (external link) program, and the now defunct Pacific Ocean Shelf Tracking (POST) program.

Much credit goes to the many ERDDAP administrators and users who have made suggestions and comments which have led to many improvements in ERDDAP. Many are mentioned by name in the List of Changes. Thank you all (named and unnamed) very much.

Here is the list of software that ERDDAP uses. We very grateful for all of these. Thank you very much. (If we have accidentally omitted a project, please let us know so we can give credit where credit is due.)

We are also very grateful for all of the software that we use when developing ERDDAP, including Chrome (external link), curl (external link), EditPlus (external link), FileZilla (external link). GitHub (external link), Google Search (external link), PuTTY (external link), the Internet, the World Wide Web, and all the great, helpful websites. Thank you very much.

License

The ERDDAP-specific code is licensed as copyrighted open source, with NOAA (external link) holding the copyright. The license is:
ERDDAP, Copyright 2016, NOAA.

PERMISSION TO USE, COPY, MODIFY, AND DISTRIBUTE THIS SOFTWARE AND ITS DOCUMENTATION FOR ANY PURPOSE AND WITHOUT FEE IS HEREBY GRANTED, PROVIDED THAT THE ABOVE COPYRIGHT NOTICE APPEAR IN ALL COPIES, THAT BOTH THE COPYRIGHT NOTICE AND THIS PERMISSION NOTICE APPEAR IN SUPPORTING DOCUMENTATION, AND THAT REDISTRIBUTIONS OF MODIFIED FORMS OF THE SOURCE OR BINARY CODE CARRY PROMINENT NOTICES STATING THAT THE ORIGINAL CODE WAS CHANGED AND THE DATE OF THE CHANGE. THIS SOFTWARE IS PROVIDED "AS IS" WITHOUT EXPRESS OR IMPLIED WARRANTY.

Contact

Questions, comments, suggestions? Please send an email to bob dot simons at noaa dot gov and include the ERDDAP URL directly related to your question or comment.

Or, you can join the ERDDAP Google Group / Mailing List by visiting https://groups.google.com/forum/#!forum/erddap (external link) and clicking on "Apply for membership". Once you are a member, you can post your question there or search to see if the question has already been asked and answered.
 


ERDDAP, Version 1.72
Disclaimers | Privacy Policy