With peer review, for example,product competitiveness, it is necessary, as in any scientific work, to carry out statistical data processing. The latter begins with the determination of the consistency of expert opinions, the numerical expression of which is the coefficient of concordance.
Why do we need an assessment of the consistency of expert opinions?
This assessment is necessary, first of all, becausethat the opinions of experts can vary greatly in the estimated parameters. Initially, the assessment is carried out on the ranking of indicators and assigning them a certain coefficient of significance (weight). Inconsistent ranking leads to the fact that these coefficients will be statistically unreliable. The opinions of experts with the required number (more than 7-10) should be distributed according to normal law.
The concept of the coefficient of concordance
So. Consistency is a concordance. The coefficient is a dimensionless quantity, showing the ratio in the general case of the variance to the maximum variance. We generalize these concepts.
The concordance coefficient is a number from 0 to 1,showing the consistency of expert opinions during the ranking of some properties. The closer this value is to 0, the consistency is considered lower. When the value of this ratio is less than 0.3, expert opinions are considered inconsistent. When finding the value of the coefficient in the range from 0.3 to 0.7, consistency is considered average. With a magnitude greater than 0.7, consistency is accepted as high.
Use cases
When conducting statistical studies canthere are situations in which an object can be characterized not by two sequences, which are statistically processed by the coefficient of concordance, but by several, which are appropriately ranked by experts who have the same level of professionalism in a certain field.
Consistency rankings implementedexperts, it is necessary to determine to confirm the correctness of the hypothesis that experts make relatively accurate measurements, which allows you to form different groups in expert groups, which are due in many respects to human factors, primarily such as differences of views, concepts, different scientific schools, the nature of professional activities, etc.
Brief description of the rank method. Its advantages and disadvantages
In the implementation of the ranking method is usedranks. Its essence lies in the fact that each property of the object is assigned a specific rank. Moreover, each expert included in the expert group is assigned this rank independently, as a result of which it becomes necessary to process this data in order to identify the consistency of expert opinions. This process is carried out by calculating the coefficient of concordance.
The main advantage of the rank method is the ease of implementation.
The main disadvantages of the method are:
- a small number of ranking objects, since if their number is exceeded by 15-20, it becomes difficult to assign objective ranking scores;
- Based on the use of this method, it remains an open question about how far in significance the objects under study are from each other.
When using this method, it is necessary to take into account that the ratings are based on a probabilistic model, therefore, they should be used with caution, taking into account the scope.
Kendall Concorde Rank Ratio
It is used to determine the relationship between the quantitative and qualitative characteristics characterizing homogeneous objects and ranked according to the same principle.
The definition of this coefficient is made by the formula:
t = 2S / (n (n-1)), where
S is the sum of the differences between the number of sequences and the number of inversions according to the second attribute;
n is the number of observations.
Algorithm of calculation:
- Ranging values x in order of either decreasing or increasing.
- Values the arranged in the order in which they correspond to the values x.
- For each subsequent rank the Determine how many rank values exceed him. They are added up and a measure of the conformity of sequences of ranks by from and and.
- Similarly, count the number of ranks. the with smaller values that also add up.
- Add up the number of ranks with exceeding values and the number of ranks with smaller values, the result is the value FROM.
This coefficient shows the relationship between the two variables, and in most cases is called the Kendall rank correlation coefficient. Such dependence can be represented graphically.
Coefficient determination
Как это делается?If the number of ranked traits or factors exceeds 2, the concordance coefficient is used, which, in its essence, is a multiple variant of the rank correlation.
Будьте внимательны.The calculation of the coefficient of concordance is based on the ratio of the deviation of the sum of squares of ranks from the average sum of squares of ranks multiplied by 12 to the square of experts multiplied by the difference between the cube of the number of objects and the number of objects.
Calculation algorithm
In order to understand where the number 12 is taken from in the numerator of the calculation formula, let's take a look at the determination algorithm.
For each row with the ranks of a particular expert, the sum of the ranks is calculated, which is a random variable.
The concordance coefficient is generally defined as the ratio of the variance estimate (D) to the maximum value of the variance estimate (Dmax). Let us give successive formulas for the definition of these quantities.
where rwed - estimation of the expectation;
m is the number of objects.
Substituting the resulting formulas in the ratio of D to Dmax we obtain the final formula for the coefficient of concordance:
Here m is the number of experts, n is the number of objects.
The first formula is used to determine the concordance rate if there are no related ranks. The second formula is used when there are related ranks.
So, the calculation of the coefficient of concordance is completed.What's next? The resulting value is estimated for significance using the Pearson coefficient by multiplying this coefficient by the number of experts and the number of degrees of freedom (m-1). The resulting criterion is compared with the tabular value, and when the value of the first exceeds the last, they indicate the significance of the coefficient under study.
In the case of related ranks, the calculation of the Pearson criterion is somewhat more complicated and is performed by the following relationship: (12S) / (d (m2+ m) - (1 / (m-1)) x (Tc1 + Tc2 + Tsn)
Example
Suppose that the expert method is estimatedthe competitiveness of butter sold in retail chains. We give an example of the calculation of the coefficient of concordance. Before assessing the competitiveness, it is necessary to rank the consumer properties of this product, which are involved in the assessment. Suppose that such properties will be the following: taste and smell, texture and appearance, color, packaging and labeling, fat content, trade name, manufacturer, price.
Let us assume that the expert group consists of 7 experts. The figure shows the results of ranking these properties.
Average value R calculated as an arithmetic average and will be 31.5. To find FROM sum the squares of the differences between Ris and R average, according to the formula given earlier, and determine that the quantity FROM is 1718.
Calculate the coefficient of concordance formula withoutthe use of related ranks (would have connected ranks if the same expert for different properties would have the same ranks).
The value of this ratio will be 0.83. This indicates a strong consistency of expert opinions.
Check its significance by the Pearson criterion:
7 x 0.83 x (8-1) = 40.7.
Pearson table test at 1% levelsignificance is 18.5, and at 5% - 14.1. Both the numbers are less than the calculated value, therefore, at a significance level of 1%, the calculated concordance coefficient is assumed to be significant.
The example demonstrates the simplicity and accessibility of the calculation for any person who has mastered the basics of mathematical calculations. To facilitate them, you can use spreadsheet forms.
Finally
Таким образом, коэффициент конкордации показывает consistency of opinions of several experts. The farther from 0 and closer to 1, the more agreed. These factors should be confirmed by calculating the Pearson criterion.
