Our data labelling tool, Argilla, is at https://argilla.labs.climatepolicyradar.org. Log in with your provided username and password.
When you log in you will see a screen with a list of tasks. We’ll use the task called rag-evaluation-axes-unece.
Once you click on the workspace, you’ll see a list of questions next to a query to the system, a generated output, and a list of sources that the LLM has had access to when generating the output. Each search is done on a single document, but details of the document are not provided as we are only interested in the model’s performance based on the sources used for the response.
Below is a guide for answering each of the questions. Ensure you consult this before you start any labelling, and feel free to refer back to it at any point during the labelling task.
Only answer each of the questions based on the text response (’Generated output’) field. E.g. for the first question, “rate the overall quality of the response”, you are rating the quality of the generated output, and not the sources or the query.
This is your subjective measure of how good the response was given the query and the provided sources.
Reminder: Only answer this question based on the quality of the text response, not the relevance of the sources.
The system is instructed to not respond in cases where there isn’t relevant information available in the sources, or where it might breach any other guidelines set for it.
There are three cases to this:
I cannot provide an answer to the question "subsidies" based on the document [0], [1], and [2].