File Name
multi_turn_agent_evaluation_with_user_simulation_metric_registration_auto_loss_analysis.ipynb
What happened?
A bug happened!
WHen running this notebook: https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/multi_turn_agent_evaluation_with_user_simulation_metric_registration_auto_loss_analysis.ipynb
Multi turn metrics are not being defined both locally and on cloud.
Loss Analysis fails completely both locally and on cloud.
Log:
Evaluation job completed in 11 seconds.
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_861893970874073088.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_8902508155590672384.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_4682072354791096320.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_2833907657708929024.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_4750189299155075072.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
Status: FAILED
Error:
code=9 details
Relevant log output
Code of Conduct
File Name
multi_turn_agent_evaluation_with_user_simulation_metric_registration_auto_loss_analysis.ipynb
What happened?
A bug happened!
WHen running this notebook: https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/multi_turn_agent_evaluation_with_user_simulation_metric_registration_auto_loss_analysis.ipynb
Multi turn metrics are not being defined both locally and on cloud.
Loss Analysis fails completely both locally and on cloud.
Log:
Evaluation job completed in 11 seconds.
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_861893970874073088.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_8902508155590672384.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_4682072354791096320.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_2833907657708929024.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
ERROR:vertexai._genai._evals_common:Failed to load evaluation result from GCS: gs://vertexsearch-447722-jul14-eval-bucket/result_4750189299155075072.json. Error: 1 validation error for EvaluationItemResult
request.candidateResponses.0.error
Extra inputs are not permitted [type=extra_forbidden, input_value={'code': 13, 'message': "...has no attribute 'get'"}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden
Status: FAILED
Error:
code=9 details
Relevant log output
Code of Conduct