CRAMP Home

Long Term Monitoring

Rapid Assessment

Mapping

Fieldwork

Accuracy Assessment
Background
Introduction
Methods
Results
Discussion
Summary
Ground Validation

Bathymetric maps

Image Interpretation

Watersheds

Publications



Search the CRAMP Website

Navigate the CRAMP Website

Bibliographic Search

Thematic Map Accuracy Assessment Results

A total of 1225 accuracy assessment (AA) field sampling stations were occupied in all four test areas combined. Three hundred ninety-three stations occurred in the Kāne‘ohe Bay test site, 304 in the Kona Coast test site, 297 in the Maui test site, and 231 in the Moloka‘i test site (Table 1). Most of these stations fell within the boundaries of the maps made from all three types of imagery. Some stations occurred in only one or two types of imagery, but not necessarily within the third imagery type. This was due in part to differences in aerial extent of imagery types at each test site.

Different imagery may have been collected on different days and under different weather conditions and sea states. Areas obscured by cloud cover, glint, waves, and turbidity differed by imagery type. These obscured areas in the imagery were mapped as the class unknown. Any sampling stations which fell within an unknown map polygon were excluded from the accuracy assessment analyses. This resulted in different numbers of stations used for analyses of each imagery type mapped. The Kāne‘ohe Bay test site had the biggest discrepancy in the numbers of stations used for analyses. While a total of 393 AA stations were occupied in the field, only 269 of these stations were used for the analyses of the map made from hyperspectral imagery, compared to 329 and 314 stations used in color aerial photography maps and IKONOS maps, respectively (Table 1).

Table 1: Total number of accuracy assessment (AA) stations occupied for each test site, broken down by imagery type and number used for final AA analyses.

 

Accuracy by Individual Test Sites

Overall Detailed-Level Accuracies

The overall map accuracies at the detailed level of the classification scheme ranged from a high of 88.5% for the maps made from color aerial photography at the Moloka‘i test site to a low of 63.4% for maps made from IKONOS imagery at the Kāne‘ohe Bay, O‘ahu test site (Table 2). Maps made from color aerial photography had higher overall accuracies than maps made from the other two types of imagery at three of the four test areas, with the exception being Kāne‘ohe Bay. At this site, maps made from hyperspectral imagery had slightly higher overall accuracy than those made from color photography, being 75.8% versus 74.5% respectively.

IKONOS maps had lower overall accuracies than maps made from the other two types of imagery at the Kona, Kāne‘ohe Bay and Maui test sites (Table 2). The IKONOS map for Moloka‘i, however, had higher overall accuracy than the hyperspectral map, but only by ~ 2%. IKONOS maps always had lower overall accuracies than maps made from color aerial photography. The Kāne‘ohe Bay IKONOS map had at least 10% less overall accuracy than any of the other types of maps for the entire state. Confidence intervals were not calculated for overall accuracies at the detailed level of the classification scheme.

Table 2: Overall Accuracies, as percentages of correctly classified sample stations, for each test area and imagery type at the detailed and major levels of the habitat classification scheme.

Overall Major-Level Accuracies

The overall accuracy at the major-level of the classification scheme ranged from a high of 96.8% for the map made from color aerial photography at the Kona test site, (with a 95% confidence interval ranging from 95.1% to 97.2%) to a low of 81.5% for the map made from IKONOS imagery at the Kāne‘ohe Bay test site (with a 95% confidence interval ranging from 78.0% to 83.3%) (Figure 1). Maps made from color aerial photography had higher overall accuracies than maps made from the other two types of imagery at three of the four test sites, with the Kāne‘ohe Bay map being the exception. The Kāne‘ohe Bay color photography map had an overall accuracy approximately one percent less than the map made from hyperspectral imagery, but was almost 5% higher than the IKONOS map for that site. Maps made from the IKONOS imagery had the lowest overall major-habitat accuracies at each of the four test sites.

Figure 1: Overall Accuracies, as percentage of correctly classified sample stations, for each test area and imagery type at the major levels of the habitat classification scheme, showing 95% confidence intervals as per Hord & Bruner (1976).

User’s and Producer’s Accuracies by Test Site

Kona

The user’s accuracy was 100% for the major habitat class “other delineations” for maps made from all three types of imagery at the Kona test site (Table 3). The user’s accuracy for the map class “unconsolidated sediments” was also 100% for the map made from hyperspectral data. The “unconsolidated sediments” class had the widest range of accuracy percentages for any habitat class, with an 82% accuracy for the IKONOS map and 100% for the hyperspectral map. No thematic map polygons were delineated for the map class “submerged Aquatic Vegetation” at the Kona test site.

The producer’s accuracy for the Kona test site ranged from a low of 83% for the “unconsolidated sediments” class using hyperspectral imagery to several 100% accuracies the in the classes “coral reef and hard bottom” and “other delineations” (Table 3). The producer’s accuracies for the class “unconsolidated sediments” were the three lowest scores for any class observed, being 87% for color photo map, 83% for the hyperspectral map, and 86% for the map made from IKONOS imagery.

Table 3: User’s and producer’s accuracies of benthic habitat map products generated from photointerpretation of major benthic habitats using color aerial photography, AURORA hyperspectral and IKONOS satellite imagery for the Kona test area.

Kāne‘ohe Bay

The user’s accuracy for the Kāne‘ohe Bay test site ranged from a low of 75% for the class “coral reef and hard bottom” using IKONOS imagery to a high of 100% for the class “other delineations” using both color aerial photography and hyperspectral imagery (Table 4). For the class “submerged aquatic vegetation”, the hyperspectral map had the lowest user’s accuracy of any of the three types of imagery. The user’s accuracies for the “other delineations” class were the highest of any classes mapped for all three types of imagery. The “unconsolidated sediments” class accuracy for the hyperspectral map was at least 10% greater than either of the other two types of imagery for this class, and was the second highest user’s accuracy observed for any type of imagery in all classes.

The producer’s accuracy for the “coral reef and hard bottom” class was the lowest of any class mapped for all three imagery types (Table 4). Within this class, the IKONOS accuracy was 67%, at least 10% lower than the other two types of imagery. The producer’s accuracy for the “other delineations” class was at least 90% or greater for any imagery type. The producer’s accuracies for the “submerged aquatic vegetation” class were very similar, varying by at most 3% for any imagery type.

Table 4: User’s and producer’s accuracies of benthic habitat map products generated from photointerpretation of major benthic habitats using color aerial photography, AURORA hyperspectral and IKONOS satellite imagery for the Kāne‘ohe Bay test area.

Maui

Compared to any other test site, the user’s accuracies for Maui varied the least between imagery types for each of the four habitat classes. There were differences of only 3% between maps made from any imagery for the “unconsolidated sediments” class, 5% for the “submerged aquatic vegetation” class, and 2% for the “coral reef and hard bottom” class. The habitat class “other delineations” did not occur in the map made from color aerial photography (Table 5).

Producer’s accuracies for the Maui test site varied the least between imagery types for each of the four habitat classes when compared to any other test site. The producer’s accuracy for “unconsolidated sediments” varied by 3% between imagery types. It ranged from 93% for color aerial photography to 90% for hyperspectral and IKONOS imagery (Table 5). The “submerged aquatic vegetation” and “coral reef and hard bottom” classes each varied by 4% between imagery types.

Table 5: User’s and producer’s accuracies of benthic habitat map products generated from photointerpretation of major benthic habitats using color aerial photography, AURORA hyperspectral and IKONOS satellite imagery for the Maui test area.

Moloka‘i

The user’s accuracy for the maps made from color aerial photography at the Moloka‘i test site were the highest among any imagery type for all four habitat classes (Table 6). The hyperspectral and IKONOS maps in the “submerged aquatic vegetation” class had the lowest user’s accuracy for the site, being a full 10% lower than the map made from color photography for this class. The “other delineations” class had 100% user’s accuracy for all three imagery types.

The producer’s accuracy was highest in three of the four habitat classes for maps made from color aerial photography, the exception being the “unconsolidated sediments” class, where IKONOS maps had higher producer’s accuracy (Table 6). The “other delineations” class had 100% producer’s accuracy for all three imagery types.

Table 6: User’s and producer’s accuracies of benthic habitat map products generated from photointerpretation of major benthic habitats using color aerial photography, AURORA hyperspectral and IKONOS satellite imagery for the Moloka‘i test area.

Kappa Statistics and Tau Coefficients

The color aerial photography maps had the highest kappa statistic ( ) of any imagery type at the Kona, Maui and Moloka‘i test sites (Table 7). At the Kāne‘ohe Bay test site, the hyperspectral map had the highest . Maps made from IKONOS imagery had the lowest at all four test sites (Table 7). The tau coefficients followed the exact same trends as that of the , however the tau values were sometimes greater than the corresponding values. The tau coefficient for the hyperspectral map at Maui was .86 compared to a of .83. For the color map at Moloka‘i, the tau was .95 compared to a of .92.

Table 7: Kappa statistics and tau coefficients of benthic habitat maps generated from photointerpretation of color aerial photography, AURORA hyperspectral and IKONOS satellite imagery at all four test sites.

Tests for Significant Differences between Tau Coefficients

There were no significant differences between the tau coefficients (Te) of maps generated from any of the three types of remotely sensed imagery at the Kāne‘ohe Bay and Maui test areas (<Zα0.05 = 1.96). At the Moloka‘i test area, maps created from color aerial photography were significantly more accurate than those created from either IKONOS (Zα0.01 = 2.622) or hyperspectral imagery (Zα0.01 = 2.374), however there was no significant difference between the hyperspectral and IKONOS imagery for this site (<Zα0.05 = 1.96). At the Kona test site, maps from color aerial photography were significantly more accurate than those based on IKONOS imagery (Zα0.05 = 2.013) but were not different from maps based on hyperspectral imagery (<Zα0.05 = 1.96). There was no significant difference between the hyperspectral and IKONOS maps at the Kona test site (Table 8).

Table 8: Summary of the z statistics indicating probability that photointerpretation of benthic habitat maps from color aerial photography, hyperspectral and IKONOS satellite imagery are equivalent based on P≤0.05 (<Zα0.05 = 1.96) with significant difference noted. The absolute value of the z statistic is compared to a standardized critical table for Z (standard normal deviate).

 

Accuracy for All Test Sites Combined

Color Photography Error Matrix

Detailed error matrices for maps interpreted from each type of imagery were combined for all test areas and aggregated into the major level of the classification scheme. The accuracy assessment sample size for all color aerial photography maps combined was 1112 (Table 9). Two hundred ninety-two of the 316 sample stations for the “unconsolidated sediments” class were mapped as “unconsolidated sediments” (they were contained within the delineated map polygons of that same class). Sixteen of the 316 sample stations were mapped as the class “coral reef and hard bottom”, and 8 were mapped as “submerged aquatic vegetation”.

For the “submerged aquatic vegetation” class, 170 of the 188 accuracy assessment sampling stations were mapped correctly by the photointerpreter (Table 9). Nine stations were mapped as “unconsolidated sediments” and another 9 were mapped as the class “coral reef and hard bottom”.

The “coral reef and hard bottom” reference stations were mapped correctly by the photointerpreter at 534 out of 559 stations. The photointerpreter incorrectly included 17 of these reference stations in polygons classified as “submerged aquatic vegetation” and 8 in the “unconsolidated sediments” class.

The photointerpreter correctly classified polygons containing 48 of the 49 reference stations for the “other delineations” class. One station was incorrectly mapped as “unconsolidated sediments”.
 

Table 9: Error matrix of accuracy assessment field data aggregated into major benthic habitat classes for maps photointerpreted from color aerial photography for all four test sites combined. Reference field data are in columns and interpreted map polygon classes are in rows. Marginal totals are for rows and columns. Bottom right cell is total of marginal values (overall sample size).
Color
Major Habitat Class
Ground Truth Major Habitats Total
Classified
A B C D
(A) Unconsolidated Sediments 292 9 8 1 310
(B) Submerged Aquatic Vegetation 8 170 17 0 195
(C) Coral Reef & Hard Bottom 16 9 534 0 559
(D) Other Delineations 0 0 0 48 48
Total Ground Truth Points 316 188 559 49 1112

Hyperspectral Imagery Error Matrix

The total number of accuracy assessment sample stations for maps made from hyperspectral imagery was 1015 (Table 10). Two hundred sixty-eight of the 296 “unconsolidated sediments” sample stations were correctly classified by the photointerpreter. Eleven stations were incorrectly mapped as “submerged aquatic vegetation” and another 17 were incorrectly classified as “coral reef and hard bottom”.

The “submerged aquatic vegetation” field data were correctly classified at 158 of the 177 sample stations. The interpreter incorrectly included 14 of these stations in the polygons for “coral reef and hard bottom” and another 5 in the “unconsolidated sediments” class. The “coral reef and hard bottom” map polygons correctly contained 455 of the 497 reference sample stations. Twenty-nine stations were incorrectly mapped as “submerged aquatic vegetation” and 13 as “unconsolidated sediments”. The photointerpreter included all 45 of the “other delineations” sample stations within the correct map class.

 

Table 10: Error matrix of accuracy assessment field data aggregated into major benthic habitat classes for maps photointerpreted AURORA hyperspectral imagery for all four test sites combined. Marginal totals are for rows and columns. Bottom right cell is total of marginal values (overall sample size).
HIS
Major Habitat Class
Ground Truth Major Habitats Total
Classified
A B C D
(A) Unconsolidated Sediments 268 5 13 0 286
(B) Submerged Aquatic Vegetation 11 158 29 0 198
(C) Coral Reef & Hard Bottom 17 14 455 0 486
(D) Other Delineations 0 0 0 45 45
Total Ground Truth Points 296 177 497 45 1015

IKONOS Imagery Error Matrix

Maps generated from IKONOS imagery contained 1068 accuracy assessment sample stations (Table 11). The “unconsolidated sediments” field data were incorrectly classified as “submerged aquatic vegetation” at 10 of the 303 stations, and another 17 were incorrectly classified as “coral reef and hard bottom”. For the “submerged aquatic vegetation” field data, 173 of the 198 stations were correctly classified. Eleven were incorrectly classified as “unconsolidated sediments” and another 14 as “coral reef and hard bottom”. “Coral reef and hard bottom” reference stations were correctly classified at 462 of the 510 stations. Twenty six were incorrectly classified as “submerged aquatic vegetation” and another 22 as “unconsolidated sediments”. The “other delineations” samples were correctly classified at 54 out of 57 stations.
 

Table 11: Error matrix of accuracy assessment field data aggregated into major benthic habitat classes for maps photointerpreted from IKONOS satellite imagery for all four test sites combined. Marginal totals are for rows and columns. Bottom right cell is total of marginal values (overall sample size).
IKONOS
Major Habitat Class
Ground Truth Major Habitats Total
Classified
A B C D
(A) Unconsolidated Sediments 275 11 22 3 311
(B) Submerged Aquatic Vegetation 10 173 26 0 209
(C) Coral Reef & Hard Bottom 17 14 462 0 493
(D) Other Delineations 1 0 0 54 55
Total Ground Truth Points 303 198 510 57 1068

Overall Detailed-Level Accuracies

The overall accuracy at the detailed level of the classification scheme for all test sites combined by imagery type ranged from a low of 74.1% for maps made from IKONOS imagery to a high of 80.8% for those made from color aerial photography (Table 12). Although the sample size for the detailed level of the classification scheme for all types of imagery combined was larger then the sample size for imagery analyzed by individual test site (Table 2), confidence intervals were not calculated at this detailed level.
 

Table 12: Overall Accuracies, as percentages of correctly classified sample stations, for each imagery type at the detailed and major levels of the habitat classification scheme.
Test Area Imagery Type Accuracy Statistics
Detailed Habitats Major Habitats
All Areas
Combined
Color 80.8 90.7
Hyperspectral 78.1 89.0
IKONOS 74.1 86.5

Overall Major-Level Accuracies

The overall accuracies at the major level of the classification scheme for each imagery type ranged from a low of 86.5% (with a 95% confidence interval ranging from 85.4% to 86.5%) for the maps made from IKONOS imagery to a high of 90.7% (with a 95% confidence interval ranging from 89.7% to 91.5%) for maps made from color aerial photography (Figure 2).

Figure 2: Overall Accuracies, as percentage of correctly classified sample stations, for all test sites combined by source imagery type at the major levels of the habitat classification scheme, showing 95% confidence intervals as per Hord & Bruner (1976).

User’s and Producer’s Accuracies

The user’s accuracy for the major habitat class “unconsolidated sediments” ranged from a low of 88.4% for maps made from IKONOS imagery to a high of 94.2% for maps made from color aerial photography (Table 13). The class “submerged aquatic vegetation” had the most variable user’s accuracies. This class also had the lowest user’s accuracies for each imagery type. User’s accuracies for the class “coral reef and hard bottom” varied by less than 5% by imagery type. The “other delineations” class had very high user’s accuracies for all three imagery types.

Producer’s accuracies for the “submerged aquatic vegetation” class were all within 3% of each other by imagery type (Table 13). For the “unconsolidated sediments” class, they were within 2%. The color aerial photography producer’s accuracy for the “coral reef and hard bottom” class was about 7% higher then the IKONOS imagery and 3% higher than the hyperspectral imagery types. The “other delineations” class ranged from 94.7% for IKONOS to 100% for hyperspectral imagery.
 

Table 13: User’s and producer’s accuracies of benthic habitat map products generated from photointerpretation of major benthic habitats using color aerial photography, AURORA hyperspectral and IKONOS satellite imagery for all test areas combined.
All Test Sites Combined Accuracy Color (%) HSI (%) IKONOS (%)
Unconsolidated Sediment User 94.2 93.7 88.4
Producer 92.4 90.5 90.8
Submerged Aquatic Vegetation User 87.2 79.8 82.8
Producer 90.4 89.3 87.4
Coral Reef and Hard Bottom User 89.3 88.9 85.6
Producer 89.3 86.9 82.8
Other Delineations User 100 100 98.2
Producer 98.0 100 94.7

Kappa Statistics and Tau Coefficients

The color aerial photography had the highest kappa statistic ( ) and tau coefficient (Te) for all test sites combined by imagery type (Table 14). The IKONOS imagery had the lowest and Te, being 0.82 and 0.83, respectively. The hyperspectral and Te were at or near the midpoint between the corresponding color and IKONOS values.
 

Table 14: Kappa statistics and tau coefficients of benthic habitat maps generated from photointerpretation of color aerial photography, AURORA hyperspectral and IKONOS satellite imagery at all test sites combined.
Test Area Imagery Type Accuracy Statistics
Kappa Tau
All Areas
Combined
Color 0.87 0.87
Hyperspectral 0.85 0.85
IKONOS 0.82 0.83

Tests for Significant Differences between Tau Coefficients

When combining data for all four test areas by imagery type, maps generated from color aerial photography were significantly more accurate than those made from IKONOS imagery (Zα0.01 = 3.071), but did not differ from hyperspectral maps (<Zα0.05 = 1.96). There was no significant difference between the accuracy of the IKONOS and hyperspectral maps (<Zα0.05 = 1.96) for all study sites combined (Table 15).

Table 15: Summary of the z statistics indicating probability that photointerpretation of benthic habitat maps from color aerial photography, hyperspectral and IKONOS satellite imagery are equivalent based on P≤0.05 (<Zα0.05 = 1.96) with significant difference noted. The absolute value of the z statistic is compared to a standardized critical table for Z (standard normal deviate).

 

Last Update: 04/21/2008

By: Lea Hollingsworth

Hawai‘i Coral Reef Assessment & Monitoring Program

Hawai‘i Institute of Marine Biology

P.O. Box 1346

Kāne‘ohe, HI 96744

808-236-7440 phone

808-236-7443 fax

email: jokiel@hawaii.edu