Quantative uncertainty scoring Quantative uncertainty scoring

"Uncertainty is not a number"

The quantification of yield gaps is subject to various sources of uncertainty, such as uncertainties in data, models, and concepts. Identification and estimation of these uncertainties provides valuable information to users of the atlas. The reported yield gaps in the atlas therefore are accompanied by qualitative estimations of uncertainties regarding the presented results, including remarks by experts on the justification of these estimations. The protocol given here describes the used approach how the uncertainty estimates are obtained. The assumption is that the GYGA framework is ‘true', i.e. it is accepted as given and there are no relevant conceptual uncertainties. Other sources of uncertainties have been framed within this conceptual view.

Terminology and explanation

Protocol parameter: phase in the GYGA protocol as described in the methods section. The relevant parameters are weather data, soil data, crop area, cropping system, modelling, and scaling.

Yield indicator: one of Yx = Yp (potential yield), Yw (water-limited yield), Ya (actual yield), or Yg (yield gap)

Level of uncertainty:

  • A qualitative estimation of the ‘size' of uncertainties (from 1 to 3 to indicate an increasing level of uncertainty) in the data used for estimating a given parameter.
  • For example, how much confidence is there in the weather data? Are there "suspicious" data?
  • The remarks section provides a concise justification of the assessment.
  • The guideline on the classification criteria for level of uncertainty given below can be used, but experts may use their own approach for estimation uncertainties if there are good reasons for this.

Sensitivity:

  • A qualitative estimation of the sensitivity of the protocol parameter (from 1 to 3 to indicate an increasing level of sensitivity), i.e. in how far is it expected that the protocol parameter relatively contributes to the end result (Yx).
    • Example: weather plays a more important role for Yw than for Yp; therefore the sensitivity to weather is lower for Yp than for Yw.
    • Example: sensitivity to rooting depth will be greater for countries with predominantly shallow soils versus countries with predominantly deeper soils.  
    • Example: In countries with low actual yields (Ya), the yield gap (Yg) is less sensitive to Ya.
 

Effect on yield:

  • A qualitative estimation of the propagated effect of the estimated uncertainty on the end result (Yx) given the sensitivity. The effect on yield is determined by using the estimated level of uncertainty (1, 2, or 3) and sensitivity (1, 2, or 3) at the hand of the look-up table given below. In this table, the levels of uncertainty (U) and sensitivity (S) have been extended to five levels to accommodate intermediate scores arising from averaging scores from two or more individual assessments. For instance, when expert A scores 1 and expert B scores 2, the average score is 1.5. In case of a divergent score (e.g. one expert scores a 1, and another a 3) this is explicitly indicated in the remarks.

Overall Uncertainty Score:

  • The "overall uncertainty score" is calculated from the values in the "effect on yield" column.
  • This score is determined by using an arithmetic mean. However, users can use the raw scores to calculate different scores if desired.
  • The overall uncertainty score is represented using the five-color-scale in the atlas, which is similar to the one given above. It is the primary information presented to the Atlas users.
 

Classification criteria for level of uncertainty

Potential yield

Protocol parameter

Criteria

Low uncertainty (1)

Medium uncertainty (2)

High uncertainty (3)

Weather

# years of measured quality data (Tmax, Tmin)

>10

3-10

0-2

 

If "suspicious data" : level -1 (please indicate why you think that the data are suspicious)

 

 

 

   

 

 

 

soil

NA

 

 

 

   

 

 

 

Crop area

observed deviations between SPAM-2005 and current land use; corrected or not; expectation there will be many more deviations (please justify this expectation)

No observed deviations after checks

Observed deviations, but corrected

Observed deviations, not corrected

   

 

 

 

Cropping system

Crop calendar: Recent (>2000); sufficient high-quality resolution (i.e. from experiments or expert networks)

Recent (>2000); high resolution

Not recent (<2000) or low resolution

Global database (not recent & low resolution)

   

 

 

 

Data for model calibration

Calibration based on recent, location-specific experimental data

Recent local high-quality experiments that allow detailed calibration of phenology and gorwth parameters

Recent local high-quality experiments (or from similar environments) that allow a limited calibration (e.g., phenology coefficients)

Calibration based on defaults model parameters or experiments of lower quality.

   

 

 

 

Scaling

Coverage (crop area in selected buffer zones / national crop area)  

>50%

25-50%

<25%

 

Also consider whether important CZ's (>5% of national area) are missing (eg if no weather data available)

 

 

 

   

 

 

 

 

Water-limited yield

Protocol parameter

Criteria

Low uncertainty (1)

Medium uncertainty (2)

High uncertainty (3)

Weather

# years of measured quality data (Prec, Tmax, Tmin, RH, Wind)

>10

3-10

1-2

 

If "suspicious data" : level -1 (please indicate why you think that the data are suspicious)

 

 

 

   

 

 

 

Soil

Resolution of the underlying database & quality of the pedotransfer functions (please justify the estimation of the resolution and the quality of the pedotransfer functions)

good resolution / PTF validated for the targeted area

moderate resolution / PTF validated for the targeted area

low resolution / PTF not validated for targeted area

 

E.g. for Europe: Confidence levels of available water content (AWC) and rooting depth (ROO)

(AWC and ROO) are "High"

(AWC or ROO) are "moderate"

(AWC or ROO) are "Low, Very Low or Unknown"

   

 

 

 

Crop area

observed deviations between SPAM-2005 and current land use; corrected or not; expectation there will be many more deviations (please justify this expectation)

No observed deviations after checks

Observed deviations, but corrected

Observed deviations, not corrected

   

 

 

 

Cropping system

Crop calender: Recent (2000); sufficient high-quality coverage (i.e. from experimental results)

Recent (>2000); high coverage

Not recent (<2000) or low coverage

Global database (not recent & low coverage)

   

 

 

 

Model

Calibration based on recent, location-specific experimental data

Recent experiments on location

Regional literature or experiments at related sites

Global literature

   

 

 

 

Scaling

Final coverage after soil selection step

>30%

10-30%

<10%

   

 

 

 

   

 

 

 

 

Actual yield

Protocol parameter

Criteria

Low uncertainty (1)

Medium uncertainty (2)

High uncertainty (3)

Yield data

Source and disaggregation by crop-water regime; if suspicious level -1 (please indicate why you think that the data are suspicious)

Crop-specific national data

Crop-aggregated national data

expert data

   

 

 

 

Number of (recent) years

# recent years for Rainfed / Irrigated

> 10 / > 5

5-10 / 3-5

<5 / <3

   

 

 

 

Scaling

Spatial resolution

District -

NUTS3 "equivalents"

Province -

NUTS2 "equivalents"

Region -

NUTS1 "equivalents"

 

Yield Gap

  • The uncertainty of Yp, Yw and Ya has been assessed separately
  • We take the average values of "uncertainty" from the Yp-, Yw- and Ya-sheets to fill in the score
  • (Yp or Yw) and Ya are given a weight to calculate the overall "effect on Yg". Currently, both components have an identical weight. The weights are given as 1, but may be adjusted by the user if there is proper justification for doing this. We are currently considering to make the weight dependent on the level of the actual yield, as uncertainty of low actual yields will have a minor effect on the uncertainty of the yield gap.

Score sheet

  • The Template is named GYGA_Uncertainty_COUNTRY_CROP.xls
    •  please rename the file for each country-crop combination (eg GYGA_Uncertainty_GHANA_MILLET.xls)
  • Grey fields to be filled (other fields are locked):
    • Four sheets for Yp, Yw, Ya and Yg
    • Country and Crop in first sheet only (Yp)
    • "Uncertainty" and "Remarks" per country-crop combination ;
    • "Sensitivity" per crop;
  • For the assessment follow the criteria given for each protocol step. Please supply concise justifications for each protocol parameter (one to three lines) in the remarks section when doing the assessment.
  • The guideline is intended not to be compulsory, but as a first estimate. If you feel that a score should be different that is perfectly acceptable as long as it is justified by proper arguments. Please note that the remarks will be explicitly included in the atlas.

 

References

Schils, R. L. M., van Voorn, G. A. K., Grassini, P., & van Ittersum, M. K. 2022). Uncertainty is more than a number or colour: Involving experts in uncertainty assessments of yield gaps. Agricultural Systems, 195.