|
 |
|
|
|
|
Synthetic Vision in the Cockpit: 3D Systems for General Aviation Andrew J. Hansen, Richard M. Rybacki and W. Garth Smith ABSTRACT Synthetic vision has the potential to improve safety in aviation through better pilot situational awareness and enhanced navigational guidance. The technological advances enabling synthetic vision are GPS based navigation (position and attitude) systems and efficient graphical systems for rendering 3D displays in the cockpit. A benefit for military, commercial, and general aviation platforms alike is the relentless drive to miniaturize computer subsystems. Processors, data storage, graphical and digital signal processing chips, RF circuitry, and bus architectures are at or out-pacing Moore's Law with the transition to mobile computing and embedded systems. The tandem of fundamental GPS navigation services such as the US FAA's Wide Area and Local Area Augmentation Systems (WAAS) and commercially viable mobile rendering systems puts synthetic vision well with the technological reach of general aviation. Given the appropriate navigational inputs, low cost and power efficient graphics solutions are capable of rendering a pilot's out-the-window view into visual databases with photo-specific imagery and geo-specific elevation and feature content. Looking beyond the single airframe, proposed aviation technologies such as ADS-B would provide a communication channel for bringing traffic information on-board and into the cockpit visually via the 3D display for additional pilot awareness. This paper gives a view of current 3D graphics system capability suitable for general aviation and presents a potential road map following the current trends. Keywords: situational awareness, mobile graphics systems, geo-specific databases, GPS navigation 1.0 INTRODUCTION The motivation for putting synthetic vision into the general aviation (GA) cockpit is primarily safety. GA pilot's typically have less flight hours and have faced fewer heavy work load situations than commercial and certainly military pilots. Synthetic vision displays show a great deal of promise for improving a pilot's situational awareness especially in poor weather conditions or reduced visibility. Initial concepts for application of synthetic vision to the general aviation cockpit are described by Barrows1. Briefly, they include the coupling of a navigation payload which includes both positioning2 and orientation3,4, with a graphical display of some digital representation of the world. Prototypes have been demonstrated at Stanford5 and Darmstadt6. Additional PC rendering concepts were presented by Hansen7 and Wagner8. A theme we develop here is the technology cross-over from other sectors such as mobile computing, image processing, and simulation. As an example, the mere idea of flying an image generator as little as 5 years ago was met with much skepticism. Indeed the thought of a refrigerator sized render engine was inconceivable. However, with the advent of PC based image generators9 the concept now appears to be only a matter of time and at pace of constant acceleration at that. We take our lead from this example. This paper focuses on the graphical rendering system needed for cockpit displays including data storage, central processing, graphics acceleration, and rendering software. Below we convey the perspective on the current status of 3D graphics systems for synthetic vision in general aviation. 2. 0 RENDERING SOFTWARE Rendering a virtual out-the-window scene for a real-time cockpit display implies a minimum of two things. There is a digital representation of the region of the world seen from the given point of view and that a system with suitable performance characteristics is available to do the rendering. The common piece in these two things is the database, which must simultaneously mimic the desired region and be efficient enough to be traversed in real-time. The motivation for the MetaVR MDX database format is to overcome performance issues related to traditional terrain database formats. The spatial organization of the MetaVR MDX greatly reduces the number of geometry sets (patches), which need to be tested prior to rendering a scene but maximizes the content in each of those patches. The MetaVR MDX format uses an advanced demand-paging scheme that eliminates the need for the database to be fully resident in memory before traversal. As with other specialized database formats, abstraction is sacrificed in the pursuit of performance. Thus the format is not particularly portable to other software applications except through the MDX API. We see the pairing of image generator and database construction as a necessity to achieve the level of performance needed for real-time rendering applications. In the rest of this section we discuss the database construction and image rendering software implemented for the express purpose of real-time image generation. 2.1 Database Construction MetaVR's WorldPerfect (WP) is a standalone Microsoft Windows application that provides the a rich, yet simple to use desktop utility for generating high resolution terrain databases from photo-realistic imagery, digital elevation data, and cultural feature content. WorldPerfect's graphical user interface (GUI) allows users to efficiently lay down super-resolution content in the form of roads, trees, rivers, buildings, and other static entities. More specifically the interface provides mechanisms for inserting runways, approach lights, taxiways, and signs. WP was specifically developed to accommodate the enormous resources needed to model large geographic regions at very high (~1m) resolution. This capability is necessary to construct the vast, detailed databases needed for aviation. During visual database design, the database engineer can draw from a broad range of source data such as DTED, DEM, and ERDAS elevation data, satellite or aerial imagery for geo-specific texturing, and vector feature data, e.g. Arc/Info and Arc/Viewer shapefiles and NIMA VMAP data.
The crux of geo-specific visual database construction is marshaling three types of source data (imagery, elevation, and features) into an efficient data structure for rendering in real-time. The main benefit to using geo-specific imagery is the added realism of the resulting 3D display. The complication is dealing with the size of the resulting database. Working from FAA circulars and approach plates, designers have the high-level functionality to control the full range of terrain creation processes from large images to the finest details on an airfield. If such detailed source is available, the designer can also work with surveys and drawing of airfield features via the shapefile import facility to bring in additional content that can be further modified if necessary. At the highest detail, WP supports manual feature layout following the geo-specific imagery. Standard database capabilities such as cut-and-fill road insertion, level-of-detail (LOD) specification, and automated culling of redundant (flat) terrain geometry are accomplished via point and click. Explicit modification of digitized source data is also available to update or correct out-of-date source data. While this is not intended to be a replacement of modeling and polygonal editing tools such as Multigen Creator, it can stand on the shoulders of such utilities by directly incorporating models generated there. Likewise, image processing applications such as Adobe Photoshop serve a role in editing specific textures such as runway markings, taxiways, building textures and light maps.
Design and construction of visual databases can be accomplished with palette based applications such as WorldPerfect. By using database generation facilities that are matched with the image generator, additional run-time performance benefits can be made. The screen capture in Figure 2 shows a completed layout of an airfield prior to compilation. Projects are stored autonomously but in a geodetic coordinate frame so that content from on project to another can be shared. Typical construction times are on the order of 2 geocells per hour given the current state-of-the-art PC. Once a database is compiled it is ready for use and is only ever parsed in a demand page fashion. Thus very large databases (~1Tb) are tractable given enough storage. The pairing of WP with the image generator does afford an additional benefit in that compression, specifically with respect to the geo-specifc imagery, can be applied right from the source to conserve storage and access resources. We turn our attention to the run-time process of rendering the out-the-window view in real-time. 2.2 Image Generation The basic task of a 3D image generator (IG) is to draw textured surfaces given a viewpoint into a visual database. For efficiency the format of the database should match as closely as possible the memory structures used in the drawing algorithms. The vast majority of PC based graphics hardware targets surfaces that are made up of polygons, and almost always triangles at that. The rasterization process is then the application of a specific texture on each triangle in the field of view. Special effects such as time of day, lighting conditions, level of detail switching and even alternate sensor modes, are implemented as modifications to that texture being applied to any given triangle. An underlying tenet of the MetaVR approach to PC image generation software is to be supported by the widest range of hardware without sacrificing performance on the top-end accelerators. One of the ways we sustained this approach was the adoption of DirectX and a Direct3D based render engine. DirectX is an integrated collection of APIs for developing media rich applications optimized for the Microsoft Windows based environment. These APIs provide low-level access to high performance 3D and 2D graphics accelerators, and sound and input devices in a device independent manner via the ``hardware emulation layer'' (HEL) and the ``hardware abstraction layer'' (HAL). These layers and APIs make it possible to run the same software on systems with or without hardware acceleration. But, more importantly, they make it possible to take advantage of advances in hardware acceleration without the need to continually re-write and re-optimize code. This has proven invaluable to MetaVR and our customers by enabling upgrades to hardware with the resultant increase in performance without the need to build and test new application code. Another area where significant performance increases can be realized is in the management of textures. Direct3D requires the application to supply textures to the graphics hardware in the pixel format and memory organization required by the hardware. This transfers a great deal of responsibility on the application developer to supply textures in the hardware-required format, with the result being that the developer is assured that the driver will not perform any expensive pixel format conversions. This feature of Direct3D has enabled our PC image generator (PC-IG) to efficiently support massive amounts of texture through a tri-tiered virtual texture memory architecture. MetaVR's commercial PC-IG, called Virtual Reality Scene Generator (VRSG), utilizes three stages of texture memory, mass storage, system memory (RAM), and texture memory resident on the graphics accelerator. Demand paging algorithms utilize the tiled LOD information encoded into the visual database to request the appropriate textures to the graphics accelerator at run-time promoting them from their current storage state if necessary. The implication here is that the IG performance is isolated from the geographic extents of the desired database and to a lesser extent the fidelity of the database being rendered. Texture that becomes sub-pixel on the display is switched out for the same but lower resolution texture. Although typically much lower resolution, the geometry representing the terrain and features is also encoded into LOD groups with successively lower and lower resolution. Demand paging is also applied to this geometry to make the performance independent of the geographic scale of the database. An exhaustive list of the IG capabilities is beyond the scope of this paper. However, some of the pertinent are non-linear terrain morphing to accommodate inter-LOD terrain switching in a graceful manner, texture fade out for intra-LOD switch outs of features, and point and lobe lighting effects for emissive and projective light sources. Some examples of real-time scenes generated with VRSG are given in the four panels below. Figure 3 is a real-time screen capture from VRSG of an area in the southwest US. The obvious terrain features, roads, towns, mountains and ridge lines, are all apparent in the scene. These visual queues are ``for free'' through the incorporation of geo-specific elevation and imagery. No manual design of cultural features is necessary. The use of geo-specific imagery throughout the database provides the important visual queues for the GA pilot such as roads, towns and rising terrain. As shown here, a town and major roads are rendered and are done so without increasing the computational complexity of either the database or the rendering process. Figure 4 is another real-time screen capture from VRSG. This screen capture depicts final approach to an airfield again in the southwest US. In this case however, the airfield features including lights, runways, runway markings, buildings, taxiways and roads were all brought into the visual database explicitly. FAA specific textures on the runway for markings and approach lights were placed in the visual database at database construction time. Figure 5 is a split-screen capture from VRSG, where on the left, wire frame mode has been turned on to illustrate the conformal cut-and-fill geometry compiled into the database to accommodate the specific airfield features. Airport specifications for such things as the runway width, length, position, orientation altitude, approach light status, taxiways as well as imagery inferred content such as buildings and roads are incorporated at database construction time. Figure 6 captures the use of a two-dimensional overlay on the out-the-window scene. Such overlays can capture standardized head up display (HUD) content or new information content specific to GPS navigation and general aviation. The overlay formats are templated and can be implemented rapidly for development and testing as well as operational displays once a design is ratified. Specific overlays such as that by Barrows1, which are more germane to general aviation, are of course needed but simple to implement given a design layout (the hard part!). 3.0 RENDERING PLATFORMS The visualization software described above is designed to support a wide range of commercial 3D graphics accelerators that can be upgraded as technology evolves. The economies of scale in PC graphics development for game applications are now doubling the performance of 3D graphics chipsets every six months. PC-IGs such as VRSG that use the Microsoft DirectX Application Programmers Interface (API) can immediately leverage this performance improvement. DirectX is a very thin driver layer above the rendering hardware. Because of the widespread support for DirectX, many of the features that originate at the driver level are pushed down into silicon at the next revision by hardware vendors. By writing to the DirectX standard application designers can take advantage of the largest possible base of potential 3D graphics accelerators. The newest generation graphics chipsets have integrated 2D and 3D capability and support visual simulation as well as desktop business graphics. 3.1 Graphics Acceleration The GeForce 2 Ultra based graphics cards (http://www.nvidia.com) are the latest generation from nVidia Corporation with a staggering peak fill rate of 1 billion bilinear filtered, multi-textured pixels per second or 2 Giga-texels per second. The Geforce 2 Ultra, shown in Figure 7 can transform, clip and light 31 million triangles per second. Each pixel pipe is capable of 7 different pixel operations in a single pass: base texture, per-pixel bump map, per-pixel diffuse lighting, per-pixel specular lighting, colored fog, ambient light and alpha transparency. These cards include integrated transform, lighting setup and rendering engines; and with 64Mb of memory the GeForce 2 Ultra will support video resolutions up to 1600x1200x32 and full scene anti-aliasing at resolutions up to 1280x1024. Maximum texture map size is up to 2048x2048-pixels. As of March 2001 the GeForce 2 Ultra has the highest fill-rate of any PC-based commercially available graphics accelerator. The Radeon 64MB DDR (http://www.ati.com) is the latest graphics card from ATI. Powered by the Radeon graphics processing unit and with 64MB of powerful DDR memory this card supports 3D resolutions up to 2048x1536x32bpp color. The Radeon, see Figure 8, provides full support for AGP 2X/4X as well as full support for and compliance with DirectX 8.0. The ATI CHARISMA ENGINE supports full transformation, clipping and lighting (T&L) at 30 million triangles/second peak processing capability. The RADEON? 3D rendering engine, powers an impressive 1.5 Giga-texels per second. The RADEON has an impressive list of 3D acceleration features including: single-pass multi-texturing, triangle setup engine, texture cache, bilinear/trilinear filtering, line and edge anti-aliasing, and full-screen anti-aliasing (FSAA). Output support is included for S-video or composite video and an integrated TMDS transmitter to support digital flat panels. Both these boards provide significant capabilities and provide different performance advantages depending upon the end application. The GeForce and Radeon cards support hardware accelerated DXTC texture compression and FSAA. Laptop solutions are also available for 3D visualization based on among others, the ATI Mobility M4 3D chip set11 with 4x AGP support. This system provides good performance (60Hz with 20,000 triangles in the field of view) at 1024x768 or 1280x1024 pixel resolution with a maximum network load of approximately 5,000 entities. With 32MB of memory the Mobility M4 of Figure 9 will support resolutions up to 1600x1200 on an external monitor and single-pass multi-texturing. The Mobility M4 is also capable of driving any 2 of 3 possible display devices simultaneously: LCD, CRT, and TV. The ATI Mobility M4 with 32MB of RAM has traditionally been fielded in the high end commercial laptop computers. Another candidate chipset is the nVidia GeForce2Go mobile that is just being introduced in the second quarter of 2001. These graphics cards are considered the minimum level of acceptable performance for rendering 2D large images and full 3D geometric environments with photo-realistic textures and cultural features. With the standardization imposed by DirectX compliance the system architect has a well-defined collection of hardware from which to design a commercial aviation package. At the high end, synthetic vision systems providing geo-specific imagery on the visual database yield a more realistic scene and presumably greater situational awareness for the pilot to discern their environment. This added realism comes at the cost of additional storage and rendering resources but does provide a major advantage over a lower end system that might use only shaded polygons or geo-typical texturing on the terrain for the visual database. The design benefit is that this spectrum can be played out against the same standardized software. Looking forward, the near-term configuration of an out-the-window synthetic vision rendering platform is very much like the Microsoft XBox game console12. The vision of using a game console as a rendering appliance does not seem far-fetched. As the development cycle accelerates with a wider base of customers and consequently a larger developer base, the aviation specific requirements for certification and robustness can be fulfilled because the underlying capability of sufficient performance for 3D visualization will exist as an embedded system. Future cars, commercial and military planes, even mobile phones will eventually have connection ports for a render appliance. 3.2 Data Storage One of the difficulties in establishing a robust and feasible 3D cockpit display has been the storage of visual databases. This is especially true where the databases utilize geo-specific imagery. As an example, in a rasterized format, 5 meter imagery requires 120Kb/km2 which on magnetic media is intractable for areas of interest much smaller than the Continental US. Further conventional random access storage such as magnetic media are too fragile for aviation use. Current advances in technology are mitigating this problem in two ways: development of new storage media and compression of the information in the visual database. The advent of large scale optical disks (DVD) and solid state flash memory provide a complementary storage solution. Vast areas of en route content can be stored on optical media. High fidelity insets, e.g. terminal areas and approach plates, can be stored on flash media. These storage devices improve robustness by reducing moving parts and efficiency by reducing weight and power consumption. Important here is that commercial solutions such as the emerging DVD standard and the prevalence of flash memory products such as the SanDisk CompactFlash and FlashDrive in Figure 10 are addressing the same sorts of storage problems. On the other hand new storage devices alone cannot manage the task. As a practical matter, commercial applications in gaming and visualization strive for improved visual quality which is synonymous with increased realism. There, the use of photo-realistic textures drove the image compression field from a pre- or post-processing step for exchanging pictures, i.e the JPEG standard, to a real-time process via wavelet compression and vector quantization of the textures in visual databases. This process has been standardized in the graphics accelerator market, enabling the texture compression discussed in Section 2. On high end ($500) consumer cardsit is burned right into the silicon for dramatic performance benefits. In this mode imagery can be propagated from source, through database compilation and demand paging to rasterization in its compressed state, thereby conserving both storage and bandwidth resources to realize significant performance improvements. For example, DirectX DXTC compression, which is a vector quantizer, achieves a fixed 8:1 compression ratio. Thus, eight times less storage and bus bandwidth is required to reconstruct an out-the-window scene for compressed texture databases compared to uncompressed raster formats available on lower-end and non-DXTC capable chipsets. 3.3 Central Processing The constrained resource requirements with regard to computational and visualization systems in aviation demand that power consumption must be kept to a minimum. To date, the two approaches to providing visual system as part of the cockpit are to either dedicate software and firmware development to special purpose low power consumption CPUs like the Intel StrongArm or the IBM PowerPC 440 or remain in an experimental stage by flying a complete development platform, in essence a desktop PC. In the former case these embedded processors do reduce power consumption but create an integration, code migration, and support path outside the traditional PC architecture. In the latter case, the modesty of the PC resource drain is remarkably superior to the dedicated IGs of the past, but not quite viable for the aviation application. However, the recent release of the Transmeta Crusoe processor13, see Figure 11 changes the baseline. The Crusoe's compatibility with standard PC architectures and low power/high performance characteristics open a new path for achieving the goal of 3D synthetic vision as a pilot's aid in the cockpit. With low power processor's that interface with existing chipsets in a ``well known'' architecture, avionics systems can retain scalability by integrating standardized subsystems such as the navigation payload or communications hardware with cross-over technologies like graphics accelerators and optical or diskless data storage devices. We believe the road map for synthetic vision in general aviation should follow a path where these new technologies leverage other state-of-the-art solutions as much as they forge elementary new capabilities. This is nowhere more true than in general aviation where the fundamental research is scarce but the grounds for improvement are rich. There is a need for added safety and the meticulous steps necessary for instituting that additional safety draw heavily on both time and dollars in the regulation process. Where possible, those resources should be dedicated to certification and training that take advantage of the technological advances available from the industrial and commercial sector. 4.0 CONCLUSIONS The reality of synthetic vision in general aviation cockpits is born of image generators that are small enough and efficient enough for an aircraft to support them. The rate of innovation in IG development will push us to the next logical step: move from a developmental ensemble of hardware to an embedded platform that is commercialized. The software migration is straightforward, the hardware migration is an exercise in integration. Databases with varying levels of fidelity have been constructed and can be stored on aviation friendly platforms. These databases carry the geo-specific content needed to aid pilots in situational awareness, navigation guidance, and in turn aviation safety. Navigation and communication data can be incorporated to drive the 3D visual display either autonomously or with awareness of external traffic when it becomes available. 5.0 REFERENCES
|
|
|
|
|
|
|
|
|
|
|
|
