Assessors Analyzer
To keep the search quality up to the mark, it is necessary that the search engine development companies regularly control and assess the output. Moreover, the assessment has to be made not by robots, but by trained professionals. The analyzer presented here is Runet's first independent project of SE assessment.
The algorithm is as follows: the search engines under assessment are given an identical set of queries; their output is collected; each query is then randomly directed to an assessor whose task is to try and guess what the searcher's intention could be and which answers would matter for him.
Since the raison d'être of this analyzer is its complete objectivity, the person assessing a specific webpage is not aware either of the search engine by which the page was yielded, or of the page's position in the output. The only thing known to him is the corresponding query.
The "Assessor's Guide" - a set of strict instructions concerning the method of assessing - allows for an even greater objectivity.
The parameters assessed are the website's relevance to the query and its quality (which is not assessed in case of an obligatory page). The first grade is a definitive one, but it can be lowered by the second grade, if two or more assessors agree that the quality of the site lies below some standardized threshold. Once the page is evaluated, its grade is multiplied by a coefficient depending of its position in the output (the higher the webpage found, the higher the coefficient). Thus we obtain a summarized grade for the whole output page. An additional parameter of evaluation for navigational queries is presence or absence of an obligatory page: its absence results in penalty points.
Once all the output pages corresponding to all the queries are evaluated, the analyzer calculates the average search quality grade for a given search engine. This is the number you can see in our informer.
 90−100%- 80−90%
- 60−80%
- 40−60%
- 20−40%
- 0−20%
|
|
