Background information about the Row Anomaly in OMI
A row anomaly is an anomaly which affects the quality of the level 1B radiance data at all wavelengths for a particular viewing direction of OMI. This corresponds to a row on the CCD dectectors, and hence the term ‘Row Anomaly’. The OMI row anomaly is dynamic, it changes over time. The row anomaly affects the quality of the Level 1B radiance data and consequently the Level 2 data products.
Please read this information carefully prior to using OMI data.
Summary on processing status
Details on the L1b post-processing status and the Level 2 re-processing status can be found in a Processing Info Document on the DISC. This document is kept up to date.
This row anomaly background information web page was updated on Friday, 26-Oct-2012 09:15:47 UTC. This web page will be updated when needed.
Feedback on row anomaly related issues (our flagging, your observations) can be sent to the OMI row anomaly team.
Overview of the dynamic behavior of the row anomaly
The row anomaly changes over time. Below we provide a short overview of the main events, but do not try to capture the complete history. A detailed account of the history of the row anomaly is kept in the lookup tables in the Level 1B software. The graphical representation displayed in figure 1 is derived from these Level 1B lookup tables. A textual representation of the same figure can be found here. Please note that the row anomaly shows slight differences between the UV2 and VIS channels, and also varies over the duration of an orbit. The graphical representation is a summary of the flags that are available in level 1B and most level 2 data products. These flags represent our best knowledge of the row anomaly. The meaning of the flags is explained at the end of this page.
- First appearance
Since June 25th of 2007 the cross-track positions 53-54 (0-based) are affected by the row anomaly. The anomaly comprises a decrease in the radiance signal, with secondary effects.
- Second group
Since May 11th of 2008 the cross-track positions 37-42 (0-based) are affected towards the northern end of the OMI orbit. The anomaly comprises a decrease or increase in the radiance signal depending on position in orbit, again with secondary effects.
On December 3rd, 2008, this anomaly expanded to cross-track position 44 (0-based) and occurs along the full length of the orbit.
- Large expansion of the row anomaly
Since January 24th, 2009 the row anomaly affects cross-track positions 27-44 (0-based). The anomaly comprises a decrease and increase in the radiance signal depending on position in orbit.
After the event of January 24th, 2009, the anomaly has continued to change more dynamically, as can be seen in figure 1, spanning many rows and at times releasing partial rows.
Since July 5th, 2011, rows 42-45 are now affected by the row anomaly over the complete orbit. These rows were already affected, but only over a partial orbit.
Since then the row anomaly shows only small changes every now and then. Consequently the row anomaly flags must then be updated in the Level 1B and Level 2 data. This is done by post-processing the level 1B data, followed by a re-processing of the Level 2 data.
Row Anomaly Flagging
The flags for the row anomaly are stored in the Level 1B data product in the new field ‘XTrackQualityFlags’. When Level 2 products are generated, this field should be copied in the Level 2 orbit data files. In addition, the Radiance Error bit or Radiance Warning bit in the Fit Quality flags or similar field of the Level 2 products should be raised, depending on the status of the Level 1B flagging. It is considered a Level 2 PI/developer responsibility to take care of this. The table below provides an overview for each product, and as of which software version and date the flagging is included.
The OMI cross track positions mentioned here are 0-based and hold for UV2 and VIS channels only. For the UV1 channel all numbers should be roughly divided by 2 and rounded down to an integer. Of cource, our Level 1B look-up tables contain the precise detailed information on which rows in the UV1 channel are affected.
Effect of the Anomaly on the OMI Radiance Spectrum
The row anomalies have four distinct effects on the OMI radiance spectra:
- 1. Blockage effect
A decrease in the radiance level for several viewing directions. It is currently assumed that this is caused by a partial blocking of the OMI nadir port. The blocking object is assumed to be opaque. This is effectively a multiplicative error on the radiances with a factor smaller than unity.
- 2. Solar radiation
An increase in the radiance level for the northern part of the orbit. This type of anomaly occurs when the part of OMI containing the nadir port is directly illuminated by the sun. This is assumed to be caused by reflection of sunlight into the nadir port via the blocking object (outside of OMI). This is an additive error on the radiances.
This increase in the radiance level is not observed for the first anomaly in rows 53-54 (0-based).
- 3. Wavelength shift
The blocking object causes an inhomogeneous illumination of the spectral slit in OMI. This causes a change in the slit function, shifting the center of weight away from the nominal center. This causes light of a specific wavelength to hit the detector in a slightly different location than expected with a fully illuminated entrance slit.
In the OMI Level 0 to Level 1 software in release 1.1.2 and before, the wavelengths are not fitted but assigned. Corrections are made based on the homogeneity of clouds and the temperature of the optical bench. Therefore the effect of an object blocking part of the incoming light is not included in the nominal level 1B wavelength assignment.
The OMI Level 0 to Level 1 software version 1.1.3 adds a field with fitted wavelengths. If these fields are used, then this third effect mostly vanishes. Developers are referred to the Level 1B documentation for details on these new fields. The documentation is linked from the DISC in the “Read me” file.
- 4. Earth radiance from outside nominal field of view
Light reflected by the earth from outside the nominal field of view is coupled into the nadir port. This light is collected over a large area, giving an additive error on the radiances, with a term which is not constant.
Effect on OMI Level 2 products
Some OMI Level 2 products rely directly on the absolute radiance levels, for instance cloud fractions (both from OMCLDO2 and OMCLDRR), aerosol properties (OMAERO and OMAERUV), nadir ozone profiles (OMO3PR), and others. These are affected by the blockage and solar radiation effects of the row anomaly. The Earth radiance from outside nominal field of view effect can have a subtle but significant effect on these products, but this is mainly a complication when developing a correction algorithm.
Other OMI Level 2 products rely on the spectral information, and are affected by the wavelength shift. This is especially true for DOAS type retrievals, such as OMCLDO2, OMNO2, OMDOAO3 and others.
Some products are affected by both wavelength shift and blockage: For example tropospheric NO2 column estimates require accurate cloud information, and are therefore affected by multiple effects of the anomaly.
The change in the radiance spectrum has a pronounced effect on the fit quality. Affected pixels in level 2 products can be recognized by significantly elevated χ2 values, stored in the ‘ChiSquaredOfFit’ field, or other fit quality parameters in the Level 2 output file. The Readme file of the product you are interested in will give advise for individual products.
Much time was spent by the OMI development team at KNMI in developing provisional solutions to mitigate the effect of the anomaly on Level 2 products. The examples below show a correction for the reduction in the radiance levels (blockage effect) and the wavelength shift. The examples are from before January 24, 2009.
Provisional solutions for the Reduced Radiance Levels (first effect)
Analysis has shown that the blocking of the entrance port depends on the position of OMI in the orbit and the wavelength. The row anomaly itself changes irregularly over time. The geophysical signal itself shows a strong seasonal cycle, complicating the analysis and corrections. Using a week of data it is possible to determine blocking factors so that corrected radiances can be produced. Figure 2 below shows the effect of the correction on the OMI O2-O2 cloud fraction.
In figure 2 some residual effect can be seen in row 41, but the anomaly at row 53 is no longer apparent. This is even more clearly shown by taking the average along the flight direction for the southern most part of an orbit, as is shown in figure 3.
A procedure to calculate radiance correction parameters has been developed. Correction parameters have been calculated and applied to the Level 1B data. Various Level 2 products have been generated using these radiance corrected Level 1B data. However, analysis shows that the accuracy of these Level 2 products is not good enough to use for scientific purposes. Therefore the effort at KNMI on developing radiance correction parameters has stopped. It may only be continued in case new information on the cause of the row anomaly becomes available.
Provisional solutions for the wavelength shift (third effect)
The wavelength of the OMI radiances can be fitted on-line (in the forward stream) with the position of the Solar Fraunhofer lines throughout the OMI wavelength range. For most Level 2 retrieval algorithms we can assume a single shift value for the respective and sufficiently narrow retrieval windows. This fitting procedure is a relatively minor change, and has been implemented in OMNO2A, OMCLDO2, and OMDOAO3 in the retrieval software version 1.1.0 and later. Figure 4 shows the fitted wavelength shift for the OMNO2A product. Note that for all row positions there is some shift required, and for the rows affected by the row anomaly the shift is about 1/10 of the OMI wavelength grid spacing.
Including this shift makes a remarkable difference to the NO2 slant columns, as is shown in figure 5. Note that radiances uncorrected for the blockage effect were used, showing that for the slant columns of NO2 the wavelength shift is the dominant effect.
The χ2 values of the fit with and without wavelength shift correction is shown in figure 6. The χ2 values show a clear reduction with the wavelength shift applied to the spectrum. A slightly higher χ2 value for the affected rows is expected, as the signal to noise ratio of the Level 1B has been reduced. Further optimizations to the wavelength grid may further improve the correction. The χ2 values of the fit without and with wavelength shift is shown in figure 6.
Several actions have already been initiated to limit the effect of the row anomaly on OMI Level 2 and higher.
A flagging field ‘XTrackQualityFlags’ has been added to the level 1B product. These flags should be carried over to the level 2 products. This change has been implemented for OMNO2, OMCLDO2 and OMDOAO3, others are to follow. Starting dates when these flags are available are given in the table.
|Product||Flagging added||Start in forward stream||Level of mitigation|
|Level 1B||Software version 1.1.2||Orbit 22510 (October 8, 2008)||XTrackQualityFlags added [note 1]|
|Incomplete flagging of the row anomaly in the XTrackQualityFlags field|
|Level 1B||Software version 1.1.3||Orbit 29515 (February 1, 2010)||Added new fitted wavelengths.|
|Row anomaly flagging is now dynamic [note 2]|
|Added orbit phase.|
|OMCLDO2||Software version 18.104.22.168||Orbit 24436 (February 17, 2009)||XTrackQualityFlags added, wavelength shift correction [note 3]|
|OMNO2A||Software version 22.214.171.124||Orbit 24349 (February 11, 2009)||XTrackQualityFlags added, wavelength shift correction [note 4]|
|OMDOAO3||Software version 1.1.1||Orbit 24583 (February 27, 2009)||XTrackQualityFlags added, wavelength shift correction [note 5]|
|OMAERO||Software version 1.1.1||Orbit 25136 (April 6, 2009)||XTrackQualityFlags added [note 6]|
- Initially these were static flags and therefore by definition not up to date. This has been corrected when all Level 1B data were post-processed.
- This means that the flagging should follow changes in the OMI row anomaly from now on. More changes were made in this version, see this OMI calibration page for a detailed description, especially the presentation linked from that page.
- The cloud fraction is not correct, as the radiance levels are uncorrected.
- The cloud fractions are not yet corrected, so the tropospheric columns are sub-optimal.
- The radiance levels are not yet corrected, so the air mass factor is still off, leading to a less accurate value of the O3 columns in the affected rows.
- The radiance levels are not yet corrected, so the aerosol properties are incorrect for the affected rows.
The meaning of the values stored in the ‘XTrackQualityFlags’ flagging field is documented in GDPS Input/Output Data Specification (IODS) Volume 2 manual, available from the OMI Level 1B DISC page. A summary is given in the table below. For Level 2 products where this field is copied, the radiance warning or error bits are set according to the value in the last column.
The value is split in two parts: bits 0-2 form a numerical value, the rest of the used bits form a set of bit-flags, indicating how a particular flag is affected. Note that before Level 1B software version 1.1.3, bits 4-7 were unused.
|Bits||Value||Meaning||Level 2 Fit flag|
|0-2||0||Not affected by row anomaly, pixel can be used|
|1||Affected by row anomaly: pixel not corrected, do not use pixel||Radiance Warning set|
|2||Somewhat affected by row anomaly: pixel not corrected, pixel can be used with caution||Radiance Warning set|
|3||Affected by row anomaly: pixel corrected, but correction is not optimal, use pixel with caution||Radiance Warning set|
|4||Affected by row anomaly: pixel corrected and correction is optimal. Pixel can be used, but is still less accurate than pixels that are not affected by row anomaly||Radiance Warning set|
|7||Affected by row anomaly. Error during the correction for the row anomaly. Do not use pixel||Radiance Error set|
|3||Reserved for future use.|
|4||Pixel may be affected by the wavelength-shift effect.|
|5||Pixel may be affected by the blockage effect.|
|6||Pixel may be affected by the stray sunlight effect.|
|7||Pixel may be affected by the stray earth radiance effect.|
Other Level 2 algorithm developers are working to copy the ‘XTrackQualityFlags’ from Level 1B to the output product, and summarize the value of this field in the fit quality flags mask as is done in OMNO2A, OMCLDO2, OMAERO and OMDOAO3. Contact the lead for the Level 2 product you are interested in to inquire about the current status if the “README” file is not clear.