Analyzer of Question Answering
This analyzer shows to what extent a search engine is able to answer queries asked in question format. These may include questions proper containing a question word ([what order is the aardvark in] [where is Lhasa located]) as well as other queries that imply a short and mostly uniform answer ([currency of Brasil] [nitric acid formula]).
The user's intent in such cases is to find such an answer. The higher its position in the search results, the better for the user. Ideally, it would appear in the first snippet or even higher, in the special result box.
This analyzer has four tabs. In each of them the total score of a search engine is the sum of its scores for each query.
1. Answer position in top 10 snippets
Here a search engine gets a score from 0 to 1 for each query, reflecting the highest position of a snippet, that contained the answer, in top 10 results. For example, if the answer was found in the first snippet, the search engine gets 1 for the query. If it was found in the second place, the score is 0.9 etc. If the answer was not found in the snippets of top 10 results, the score is 0.
2. Answer presence in snippets
Here a search engine gets 1 for a query if at least one of the top 10 snippets contained an answer to the question. 0 is assigned otherwise.
3. Position of answer web pages in top 10
A search engine gets a score from 0 to 1 for each query, reflecting the highest position of a web page, that contained the answer, in top 10 results. The score is 1 if the first page found contained an answer, 0.9 if the answer was present on the second page and 0 if none of the web pages found contained the answer.
4. Answer presence on web pages found
Here a search engine gets 1 for a query if at least one of the top 10 web pages contained an answer to the question. 0 is assigned otherwise.
For some queries the correct answer may have a few variants. For example, the query 'Olympic motto' is adequately answered whether the motto comes in Latin, in English or in the native language of the user. As long as these are variants of the same entity, we count them as correct. But we avoided the queries with multiple possible answers (like 'who was the wife of Henry the Eighth') as improper for this analyzer.