ERDDAP:
Heavy Loads, Grids, Clusters, Federations,
and Cloud Computing

Heavy Loads / Constraints

1. A remote data source's bandwidth - Unless a remote data source has a very high bandwidth Internet connection, ERDDAP's responses will be constrained by how fast ERDDAP can get data from the data source. A solution is to copy the dataset onto ERDDAP's hard drive, perhaps with EDDGridCopy or EDDTableCopy.
2. ERDDAP's server's bandwidth - Unless ERDDAP's server has a very high bandwidth Internet connection, ERDDAP's responses will be constrained by how fast ERDDAP can get data from the data sources and how fast ERDDAP can return data to the clients. The only solution is to get a faster Internet connection.
3. Memory - If there are many simultaneous requests, ERDDAP can run out of memory and temporarily refuse new requests. (ERDDAP has a couple of mechanisms to avoid this and to minimize the consequences if it does happen.) So the more memory in the server the better. On a 32-bit server, 4+ GB is really good, 2 GB is okay, less is not recommended. On a 64-bit server, you can almost entirely avoid the problem but getting lots of memory. See the -Xmx and -Xms settings for ERDDAP/Tomcat.
4. Hard drive bandwidth - Accessing data stored on the server's hard drive is vastly faster than accessing remote data. Even so, if the ERDDAP server has a very high bandwidth connection, it is possible that accessing data on the hard drive will be a bottleneck. A partial solution is to use fast (e.g., 10,000 RPM) hard drives.
5. Too many files in a cache directory - ERDDAP caches all images, but only caches the data for certain types of data requests. It is possible for the cache directory for a dataset to have a large number of files temporarily. This will slow down requests to see if a file is in the cache (really!). <cacheMinutes> in setup.xml lets you set how long a file can be in the cache before it is deleted. Setting a smaller number would minimize this problem.
6. CPU - Making graphs is the only thing that takes significant CPU time (roughly 1 second per graph). So if there were many simultaneous unique requests for graphs (WMS clients!), there could be a CPU limitation. On a multi-core server, it would take a lot of requests before this became a problem.
    

Grids, Clusters, and Federations

We're using grid as a general term to indicate a type of computer cluster where all of the parts may or may not be physically located in one facility and may or may not be centrally administered. An advantage of co-located, centrally owned and administered grids (clusters) is that they benefit from economies of scale (especially the human workload) and simplify making the parts of the system work well together. An advantage of non-co-located grids, non-centrally owned and administered (federations) is that they distribute the human work load and the cost, and may provide some additional fault tolerance. The solution we propose below works well for all grid topographies.

The basic idea of designing a scalable system is to identify the potential bottlenecks and then design the system so that parts of the system can be replicated as needed to alleviate the bottlenecks. Ideally, each replicated part increases the capacity of that part of the system linearly (efficiency of scaling). The system isn't scalable unless there is a scalable solution for every bottleneck. Scalability is different from efficiency (how quickly a task can be done -- efficiency of the parts). Scalability allows the system to grow to handle any level of demand. Efficiency (of scaling and of the parts) determines how may servers, etc., will be needed to meet a given level of demand. Efficiency is very important, but always has limits. Scalability is the only practical solution to building a system that can handle very heavy use. Ideally, the system will be scalable and efficient.

The goals of this design are:

• To make a scalable architecture (one that is easily extensible by replicating any part that becomes over-burdened). To make an efficient system that maximizes the availability and throughput of the data given the available computing resources. (Cost is almost always an issue.)
• To balance the capabilities of the parts of the system so that one part of the system won't overwhelm another part.
• To make a simple architecture so that the system is easy to set up and administer.
• To make an architecture that works well with all grid topographies.
• To make a system that fails gracefully and in a limited way if any part becomes over-burdened. (The time required to copy a large datasets will always limit the system's ability to deal with sudden increases in the demand for a specific dataset.)
• (If possible) To make an architecture that isn't tied any specific cloud computing service or other external services (because it doesn't need them).

Our recommendations are:

• Basically, we suggest setting up a Composite ERDDAP (D in the diagram), which is a regular ERDDAP except that it just serves data from other ERDDAPs. The grid's architecture is designed to shift as much work as possible (CPU usage, memory usage, bandwidth usage) from the Composite ERDDAP to the other ERDDAPs.
• ERDDAP has two special dataset types, EDDGridFromErddap and EDDTableFromErddap, which refer to
  datasets on other ERDDAPs.
• When the composite ERDDAP receives a request for data or images from these datasets, the composite ERDDAP redirects the data request to the other ERDDAP server. The result is:
  • This is very efficient (CPU, memory, and bandwidth), because otherwise
    1. The composite ERDDAP has to send the data request to the other ERDDAP.
    2. The other ERDDAP has to get the data, reformat it, and transmit the data to the composite ERDDAP.
    3. The composite ERDDAP has to receive the data (using extra bandwidth), reformat it (using extra CPU time and memory), and transmit the data to the user (using extra bandwidth).
    By redirecting the data request and allowing the other ERDDAP to send the response directly to the user, the composite ERDDAP spends essentially no CPU time, memory, or bandwidth on data requests.
  • The redirect is transparent to the user regardless of the client software (a browser or any other software or command line tool).

The parts of the grid are:

A) For every ERDDAP data source that has a high-bandwidth server, use EDDGridFromErddap or EDDTableFromERDDAP to serve the data in the Composite ERDDAP.

B) For every ERDDAP-able data source (a data source from which ERDDAP can read data) that has a high-bandwidth server, set up another ERDDAP in the grid which is responsible for serving the data from this data source.

• If several such ERDDAPs aren't getting many requests for data, you can consolidate them into one ERDDAP.
• If the ERDDAP dedicated to getting data from one remote source is getting too many requests, there is a temptation to add additional ERDDAPs to access the remote data source. In special cases this may make sense, but it is more likely that this will overwhelm the remote data source (which is self-defeating) and also prevent other users from accessing the remote data source (which isn't nice). In such a case, consider setting up another ERDDAP to serve that one dataset and copy the dataset on that ERDDAP's hard drive (see C) using EDDGridCopy and/or EDDTableCopy.

C) For every ERDDAP-able data source that has a low-bandwidth server (or is a slow service for other reasons), consider setting up another ERDDAP and storing a copy of the dataset on that ERDDAP's hard drives using EDDGridCopy and/or EDDTableCopy. If several such ERDDAPs aren't getting many requests for data, you can consolidate them into one ERDDAP.

D) The composite ERDDAP is a regular ERDDAP except that it just serves data from other ERDDAPs.

• Because the composite ERDDAP has information in memory about all of the datasets, it can quickly respond to requests for lists of datasets (full text searches, category searches, the list of all datasets), and requests for an individual dataset's Data Access Form, Make A Graph form, or WMS info page. These are all small, dynamically generated, HTML pages based on information which is held in memory. So the responses are very fast.
• Because requests for actual data are quickly redirected to the other ERDDAPs, the composite ERDDAP can quickly respond to requests for actual data without using any CPU time, memory, or bandwidth.
• By shifting as much work as possible (CPU, memory, bandwidth) from the Composite ERDDAP to the other ERDDAPs, the composite ERDDAP can appear to serve data from all of the datasets and yet still keep up with very large numbers of data requests from a large number of users.
• Preliminary tests indicate that the composite ERDDAP can respond to any request in ~1ms of CPU time, or 1000 requests/second. So an 8 core processor should be able to respond to about 8000 requests/second. Although it is possible to envision bursts of higher activity which would cause slowdowns, that is a lot of throughput. It is likely that data center bandwidth will be the bottleneck long before the composite ERDDAP becomes the bottleneck.
• In very, very extreme cases, you may want to set up more than one composite ERDDAP. It is likely that other parts of the system (notably, the data center's bandwidth) will become a problem long before the composite ERDDAP becomes a bottleneck. So the solution is probably to set up additional, geographically diverse, data centers (mirrors), each with one composite ERDDAP and servers with ERDDAPs and (at least) mirror copies of the datasets which are in high demand. Such a setup also provides fault tolerance and data backup (via copying).
• [For a fascinating design of a high performance system running on one server, see Cringely's overview or the detailed description of Mailinator.]

Datasets In Very High Demand - In the really unusual case that one of the A, B, or C ERDDAPs can't keep up with the requests because of bandwidth or hard drive limitations, it makes sense to copy the data (again) on to another server+hardDrive+ERDDAP, using EDDGridCopy and/or EDDTableCopy. While it may seem ideal to have the original dataset and the copied dataset appear seamlessly as one dataset in the composite ERDDAP, this is difficult because the two datasets will be in slightly different states at different times (notably, after the original gets new data, but before the copied dataset gets its copy). Therefore, we recommend that the datasets be given slightly different titles (e.g., "... (copy #1)" and "... (copy #2)", or perhaps "(mirror #n)" or "(server #n)") and appear as separate datasets in the composite ERDDAP. Users are used to seeing lists of mirror sites at popular file download sites, so this shouldn't surprise or disappoint them. Because of bandwidth limitations at a given site, it may make sense to have the mirror located at another site. If the mirror copy is at a different data center, accessed just by that data center's composite ERDDAP, the different titles (e.g., "mirror #1) aren't necessary.

RAIDs vs. Regular Hard Drives - If a large dataset or a group of datasets are not heavily used, it may make sense to store the data on a RAID since it offers fault tolerance and since you don't need the processing power or bandwidth of another server. But if a dataset is heavily used, it may make more sense to copy the data on another server + ERDDAP + hard drive (similar to what Google does) rather than to use one server and a RAID to store multiple datasets since you get to use both server+hardDrive+ERDDAPs in the grid until one of them fails.

Failures - What happens if...

• There is a burst of requests for one dataset (e.g., all students in a class simultaneously request similar data)? Only the ERDDAP serving that dataset will be overwhelmed and slow down or refuse requests. The composite ERDDAP and other ERDDAPs won't be affected. Since the limiting factor for a given dataset within the system is the hard drive with the data (not ERDDAP), the only solution (not immediate) is to make a copy of the dataset on a different server+hardDrive+ERDDAP.
• An A, B, or C ERDDAP fails (e.g., hard drive failure)? Only the dataset(s) served by that ERDDAP are affected. If the dataset(s) is mirrored on another server+hardDrive+ERDDAP, the effect is minimal. If the problem is a hard drive failure in a level 5 or 6 RAID, you just replace the drive and have the RAID rebuild the data on the drive.
• The composite ERDDAP fails? If you have no hot or warm failover, the system is effectively down until you set up a replacement. That's not good. So if this is a concern, set up multiple composite ERDDAPs or a hot or warm failover. Then the effect is minimal. Failures of the composite ERDDAP should be rare because it has minimal hard drive activity.

Simple, Scalable - This system is easy to set up and administer, and easily extensible when any part of it becomes over-burdened. The only real limitation for a given data center is the data center's bandwidth.

Bandwidth - Note the approximate bandwidth of commonly used components of the system:

• Encourage clients to request subsets of the data if that is all that is needed. If the client only needs data for a small region or at a lower resolution, that is what they should request. Subsetting is a central focus of the protocols ERDDAP supports for requesting data.
• Encourage transmitting compressed data. ERDDAP compresses a data transmission if it finds "accept-encoding" in the HTTP GET request header. All web browsers use "accept-encoding" and automatically decompress the response. Other clients (e.g., computer programs) have to use it explicitly.
• Colocate your servers at an ISP or other site that offers relatively less expensive bandwidth costs.
• Disperse the servers with the ERDDAPs to different institutions so that the costs are dispersed. You can then link your composite ERDDAP to their ERDDAPs.

For general information on designing scalable, high capacity, fault-tolerant systems, see Michael T. Nygard's book Release It.

[Yes, these are simplistic calculations, but I think the conclusions are correct. Did I use faulty logic or make a mistake in my calculations? If so, the fault is mine alone. Please send an email with the correction to bob dot simons at noaa dot gov.]
 

Cloud Computing

• They offer access to very high bandwidth Internet connections. This alone may justify using these services.
• They only charge for the services you use. For example, you get access to a very high bandwidth Internet connection, but you only pay for actual data transferred. That lets you build a system that rarely gets overwhelmed (even at peak demand), without have to pay for capacity that is rarely used.
• They are easily extensible.
• They free you from many of the administrative duties of running the servers and networks.

• They charge for their services, sometimes a lot (in absolute terms; not that it isn't a good value). The prices listed here are for Amazon EC2. These prices (as of March 2009) will come down, but there will also probably be more data and more requests for data in the future.
  • Data Transfer - $0.10/GB for data transfers into the system and $0.10/GB for data transfers out of the system. Storing 1 TB in the cloud will cost $100 for the initial data transfer fees alone. But the ongoing cost for data transfer is the bigger problem. One SATA hard drive (0.3GB/s) on one server with one ERDDAP can probably saturate a Gigabit Ethernet LAN (0.1GB/s). And one Gigabit Ethernet LAN (0.1GB/s) can probably saturate an OC-12 Internet connection (0.06GB/s). If one OC-12 connection can transmit ~150,000 GB/month, the Data Transfer costs will be about 150,000 GB @ $0.10/GB = $15,000/month. Clearly, if you have a dozen hard-working ERDDAP's on a cloud service, your monthly Data Transfer fees could be substantial (up to $180,000/month). (Again, it isn't that the service is over-priced, it is that we are buying a lot of the service.)
  • Data storage - Amazon charges $150/month per TB. (Compare that to buying a drive outright for ~$150/TB). So if you need to store lots of data in the cloud, it might it might be fairly expensive (e.g., 20TB would cost $3000/month). But this is a minor issue compared to bandwidth/data transfer costs.
• The subsetting problem: The only way to efficiently distribute data which is stored in the cloud, is to have the program which is distributing the data (ERDDAP) running on a server which has the data stored on a local hard drive (or a SAN). Many data requests (notably gridded data requests where the stride value is > 1) can't be done efficiently if the program has to requests big chunks of a dataset from a non-local data storage system and then extract a subset.

Thanks - Many thanks to Matthew Arrott and his group for their work on putting ERDDAP in the cloud and the resulting discussions.

Contact

Questions, comments, suggestions? Please send an email to bob dot simons at noaa dot gov and include the ERDDAP URL directly related to your question or comment.
 

ERDDAP, Version 1.42
Disclaimers | Privacy Policy