DeepEval Test Results

evalReport.json · uploaded 11/14/2026, 12:25:27 AM · exported 11/14/2026, 1:12:05 AM

Overall evaluation score

0.69 / 1.0

DeepEval execution status

TESTS FAILED

Mean of 5 metric score(s) across 1 test case(s), each result weighted equally. Metric results: 3 passed, 2 failed.

Total tests1
Passed0
Failed1
Duration3m 13.4s
Eval cost$0.0000
Tools executed1

Metric analytics

MetricAvg scoreThresholdScorePassFail
Task Completion0.900.70
10
Tool Correctness1.000.70
10
Answer Relevancy1.000.70
10
Output Correctness [GEval]0.200.70
01
Faithfulness0.330.70
01

Test cases

test_weather_metrics_with_llama31

FAIL

Duration 3m 12.4s · Cost $0.0000

Input

What is the current weather in Tokyo?

Actual output

The tool output indicates that the current temperature in Tokyo is 17.68°C, with broken clouds and 59% humidity.

Expected output

The current weather in Tokyo is 22.1°C with clear skies and 67% humidity.

Metrics

MetricScoreThresholdResultReason
Task Completion0.900.70PASS The system provided the current weather in Tokyo, including temperature, cloud conditions, and humidity, which aligns with the desired task of providing the current weather information.
Tool Correctness1.000.70PASS [ Tool Calling Reason: Exact match: expected ['get_weather_prediction'], called ['get_weather_prediction']. See details above. Tool Selection Reason: The agent selected the 'get_weather_prediction' tool, which is perfectly suited for the task of getting the current weather in Tokyo. The tool's description matches the task's requirements, and it was used correctly with the necessary input parameter 'location' set to 'Tokyo'. ]
Answer Relevancy1.000.70PASS The score is 1.00 because the actual output directly answers the question about the current weather in Tokyo, making it highly relevant.
Output Correctness [GEval]0.200.70FAIL The Actual Output does not match the Expected Output in terms of temperature, weather conditions, and humidity. The temperature is off by 4.42°C, the weather conditions are different, and the humidity is off by 8%. The Actual Output also reports the correct location (Tokyo).
Faithfulness0.330.70FAIL The score is 0.33 because the actual output contradicts the retrieval context in two instances: it claims the current temperature in Tokyo is not 22.1°C and the weather is not clear, indicating a lack of faithfulness to the original context.

Tool calls

ExpectedActualResult
get_weather_prediction
{
  "location": "Tokyo"
}
get_weather_prediction
{
  "location": "Tokyo"
}
MATCH