JES Builds Simulation Storage Systems for the A-10 Program Using FIBRE SAN Technology for VSRG Applications

MetaVR VSRG technology has enabled the use of COTS PC-IG and high-speed video adapters which in turn has instigated a proliferation of relatively inexpensive simulator applications. Engineers have begun taking advantage of this technology to the point where they want high-speed synchronized systems with increasingly more PC-IG channels than what was ever anticipated.

Customers might require 25, 50, or more channel simulators using VSRG for different reasons; some have several simulators using a single data source at a training center. Others have a large aircraft squadron that interacts with a ground unit. Others want to tie together a large convoy of Humvees or a fleet of ships. Using a FIBRE infrastructure in simulators running MetaVR VSRG can significantly improve performance and increase data storage capacities.

View from the cockpit of an A-10 simulator using MetaVR multi-channel visual systems.

JES Hardware Solutions provides high speed storage to the A-10 program which needed to improve performance and provide a large data pool on an aircraft simulator. Working side-by-side with engineers, JES developed a new high speed FIBRE infrastructure and a larger SATA II FIBRE RAID system to increase overall simulator performance.

This JES case study describes two multichannel IG configurations: a simple FIBRE upgrade to drive 11 PC-IG channels and a more complex upgrade to support simulators that can use 25,35, or 100 PC-IG channels. This study, which describes read-only solutions, includes various security issues that FIBRE solves, such as distance and the secure nature of FIBRE verses wire. (In a read-only solution, a system manager can only make changes to terrain databases; no writing of data can take place while simulations are running.) All systems unless otherwise noted are fully deployed or past the proof-of-concept phase.

Simple FIBRE infrastructure for an 11-channel simulator

The diagram below shows a simple FIBRE infrastructure for an 11-channel simulator. Each IG channel has a single-channel PCI Express 4G FIBRE adapter.

The use of PCI-Express FIBRE channel cards is recommended as the PCI Express bus will take full advantage of 4G speed, however upgrades can use other types of PCI cards and will take advantage of the FIBRE performance increase. All IG channels are attached to a 4G FIBRE switch. The switch does not have to be an expensive director class switch, just a 4G switch, such as Qlogic 5600 4GB switch. This unit has a single power supply. Redundant PS systems are possible though are not really necessary on the smaller simulators. The simulation data is stored on a 16-bay dual 4G host JES RAID system. This RAID system can hold up to 4GB of cache RAM, but at the time of this writing only a 2GB chip is available. This system uses 500GB enterprise level SATA II drives. SATA II drives have capacities of up to 1TB, however 500GB drives are recommended because the RAID will perform faster with more drives present. The RAID system is segmented into two drives. This system uses 14 drives to make a level 5 RAID data protected 6TB volume and a 1TB RAID 0 unprotected drive pair. This configuration allows the terrain data to remain on the protected disk and the simulation database data to be on a fast swappable drive pair; this in turn allows for the easy changing of simulation databases using the same terrain data. In summary, this system is a RAW FIBRE storage system. The RAID connects directly to the FIBRE switch, which tricks each PC-IG channel into thinking the drive is local. There is an open 4G PIPE to each PC-IG channel. With JES assistance on PC-IG and RAID setup, this upgrade was easily achieved for all ten simulators in the field. The prime contractor team working on the simulator pictured here commissioned JES to build a 200TB mirrored 4G FIBRE SAN system, which will eventually grow to several PB (1000TB) over time. With the SAN system installed, simulations take less than one day to compile; thus simulations can be changed easily and deployed quickly in this non-proprietary COTS design.

Infrastructure for a high-speed 21-channel simulator for training and mission rehearsal

After introducing a FIBRE infrastructure to simulator design, JES was commissioned to design a high-speed 21-channel simulator for training and mission rehearsal. Again the reason for the SAN-like system was the need for a large centrally located storage pool and the demonstrated high speed of FIBRE. This system gives a 21-channel synchronized panoramic (360 degree) view of the virtual world. However 21, 35, 50 or even 100 synchronized channels can have many different applications.

As illustrated in the diagram below, this system contains several changes from the simple raw FIBRE system. First, it uses three switches of the high availability type; each switch has 10 active ports.

Next, it uses a SAN control computer, which is currently the fastest COTS computer available using dual processors. The computer is then filled to maximum with dual port FIBRE channel adapters. (Quad port adapters could be used but they oversaturate x16 PCI express slots and the dual channel is just right.) Windows 2003 Advance Server is used and the motherboard is filled with 32GB of RAM. This provides a large cache RAM to buffer and complete storage requests. The SAN control software really brings the performance to this system. The 32GB cache RAM allows for quick dispersal of read requests and takes advantage of larger SATA II disks. The software enables assignment of FIBRE channels to separate switches as noted above in the diagram. Also in the diagram are two 4G FIBRE channels leading to each 7 PC-IG cluster. The SAN control system is the heart of the performance needed to drive 21 or more channels from a single data source in perfect synchronization.

The storage system is JES standard SATA II to 4G FIBRE system, yielding 10TB at level 5 with hot spare. Again both host channels are plugged into the SAN control computer. Using both host channels enables maximizing the 2GB cache RAM inside RAID system. There are enough FIBRE ports to add several more RAID systems if more storage is needed.

A requirement for this particular system was the ability to boot the PC-IG channels remotely. Remote boot over FIBRE is at best unreliable. It is better to remote boot down Gigabit Ethernet (GbE) than to clog the FIBRE with PC-IG commands; all computers purchased today come with at least two GbE ports. JES uses remote boot software that works very well; you set up the PC-IG channel in the way you want it and take an image of it. The system works via ISCSI down GbE using ISCSI block transfer protocol, which enables the fast remote booting of many systems. The benefit of this remote boot software is that if a PC-IG channel fails, you can simply plug in a new computer; the channel will return right back to it pre-failed state. As you can see from the diagram above, the remote boot system is on its own subnet.

The design of a 21-channel simulator using COTS products to create a GOTS/COTS simulator took almost a full year to complete. After the system was successfully implemented and tested, JES moved from proof-of-concept to the full deployment of 52 systems. The system still has the basic concepts of the simple FIBRE 11-channel system; the key is essentially tricking PC-IG channels into thinking that the storage is local. The proof of concept driving this many VSRG channels has prompted requests for all types of applications from 25 to 100 channels of PC-IG using VSRG.

This 21-channel system has a great deal of potential and has really yet to be fully utilized. To understand the potential of FIBRE channel SAN topology, consider that a 100-channel design has 25TB of mirrored data and 20 FIBRE lines over ten switches using two mirrored SAN control computers over ISCSI.

Security measures and other modifications

There are two types of FIBRE -- copper and glass. Copper FIBRE lines are inexpensive but can only be used for short distances; they emit radiation, which often makes them unsuitable for meeting security requirements. Glass FIBRE lines can extend for thousands of meters; they are composed of light. Some of JES' customers require a very long extension of FIBRE lines in order to have data locked in a secure area. In such a case an armored cable composed of several FIBRE lines is used. An armored cable protects the glass FIBRE over long distances. This scenario is an improvement over having to take 10, 35, 50 or even 100 local hard drives out of each machine at the end of the day. It is certainly more efficient and, in the long run, a more cost-effective method of delivering secure data to a simulator. JES has also designed systems that enable users to detach the storage and SAN control from the simulator to transport the system to a secure area at the end of the day.

Adding FIBRE technologies to COTS simulator design using MetaVR VSRG can significantly improve performance, enlarge the scope of work, and increase storage capacities. The introduction of FIBRE topologies has essentially returned the bottleneck of the simulator back to the video card.

Information and diagrams on this page courtesy of James Sessions, President of JES Hardware Solutions.