wiki:cookbook/QCOMP
Last modified 3 years ago Last modified on 09/15/14 18:19:13

1. Algorithm name

Quality-based compositing: QCOMP

2. Basic description

Based on existing quality controls in the BALTRAD Toolbox, tools improving and characterizing data quality are selected and chained together to produce a QI_TOTAL quality index that is used as the prioritized compositing criterion.

a) Physical basis of the algorithm

"The" algorithm is in fact a combination of several. It is not necessarily physical, but rather quality-based which is inherently a combination of both quantitative (ropo, hac, beamb, radvol-broad) and qualitative aspects (radvol-broad, poo) of QC.

Each polar QC step is performed prior to composite generation. QI_TOTAL is calculated for each site, and then it is used as the prioritized compositing criterion. The composting itself is performed using the toolbox compositing functionality that navigates input polar data directly to the output composite Cartesian surface without any intermediate transformation(s).

b) Amount of validation performed so far

Several BALTRAD partners are running ropo operationally, and both ropo and hac are running operationally at Odyssey. beamb was validated in early 2012 as part of the trial of the BALTRAD toolbox for Odyssey. RADVOL has been validated by IMGW. poo is being used operationally at FMI. QI_TOTAL has not yet been validated, alone or in compositing.

c) References (names and contact information of all developers during the evolutionary history, scientific papers)

QC Reference Description
ropo BALTRAD Manual Anomaly detection and removal. Operational at Odyssey. Descriptive and corrective.
hac BALTRAD Cookbook Hit-accumulation clutter filtering. Operational at Odyssey. Descriptive and corrective.
beamb BALTRAD Cookbook Beam-blockage analysis and correction. Descriptive and corrective.
radvol-att BALTRAD Cookbook Conventional single-polarization attenuation correction. Descriptive and corrective.
radvol-broad BALTRAD Cookbook Quality characterization due to distance to radar related effects. Overall analysis of range and pulse volume. Descriptive.
poo BALTRAD Cookbook Probability of overshooting. Descriptive.
QI_TOTAL BALTRAD Cookbook "Total" quality. Descriptive.

Additional references:

Koistinen J. and Hohti H., 2010: Operational diagnosis of precipitation detection range. Proc. ERAD 2010 (short abstract).

Michelson D. and Henja A., 2012: OPERA Work Package 3.6: Odyssey additions. Task 3. Tuning and evaluation of “andre” tool. EUMETNET OPERA Working document WD_2012_02c. 20 pp.

Henja A. and Michelson D., 2012: Improving the quality of European weather radar composites with the BALTRAD toolbox. Proc. ERAD 2012. Météo France, Toulouse, France

Michelson D. and Henja A., 2013: OPERA WP3.6, Task #4. Implementation of hit-accumulation clutter filter in BALTRAD toolbox. EUMETNET OPERA Working document WD_2012_02p. 8 pp.

Ośródka, K., Szturc, J., and Jurczyk, A., 2012. Chain of data quality algorithms for 3-D single-polarization radar reflectivity (RADVOL-QC system). Meteorol. Appl.

Peura M., 2002: Computer vision methods for anomaly removal. Proc. ERAD 2002, 312–317

3. ODIM metadata requirements for I/O

Limited optional 'how' metadata are required in processing polar data: RADVOL-BROAD requires the beamwidth and pulsewidth. Otherwise, only mandatory metadata from 'what' and 'where' groups. In generating the composite, metadata and quality fields generated from the polar quality processing chain are used.

4. Input data

a) What kind of radar data (including the list of previous algorithms and quality flags applied)

Horizontally-polarized reflectivity (DBZH) is the basis for the polar QC chain and QCOMP.

b) Other data (optional and mandatory, applying “universally” agreed formats, geometry)

5. Logical steps

GraphViz image

Chaining the polar QC algorithms together is a simple matter of using a script like odc_toolbox on the command line:

$ odc_toolbox -i input_directory -o output_directory -q hac-increment,ropo,hac-filter,beamb,radvol-att,radvol-broad,poo,qi_total

In generating QI_TOTAL for each site, each of its input quality indicators can be assigned a weight. These can be looked up from XML file; default weights are included.

Composite generation using QCOMP is performed using the output from polar QC. The compositing must be based on a conventional selection criterion, e.g. in cases where all input data have the same quality. This criterion is selectable from among those presently available in the toolbox 'slow compositing' functionality: nearest radar, lowest value to the Earth's surface, PMAX and MAX. Each site's QI_TOTAL may be assigned a weight when generating the composite. This can be looked up from the same XML file as is used to process the single-site polar input data. A multiplicative model is suggested for calculating QI_TOTAL for these purposes.

Performance of this algorithm can be greatly improved if this composite generator is "tiled" for large composite domains, as was successfully shown in the trial of the toolbox for Odyssey (Michelson and Henja, 2013, Henja and Michelson, 2012). A prerequisite, however, is that there are as many logical CPU cores as there are tiles. For the trial at the European level, six logical CPU cores were required. (A normal quad-core CPU has eight logical cores.)

6. Output.

a) Data type using ODIM notation where possible, e.g. DBZH

b) Added quality indicators

QC /how/task Comment
ropo fi.fmi.ropo.detector.classification
hac eu.opera.odyssey.hac
beamb se.smhi.detector.beamblockage
radvol-att pl.imgw.radvolqc.att
radvol-broad pl.imgw.radvolqc.broad
poo se.smhi.detector.poo
qi_total pl.imgw.detector.qi_total
distance se.smhi.composite.distance.radar Optionally added during composite generation

7. Outline of a test concept exemplifying the algorithm, as a suggestion for checking that an implementation has been successful.

The formulation of unit test(s) is TBD.

The trial of this implementation can be carried out on one or several months of data and validated against gauges as was done with the trial of ropo and beamb.