Summary
RoleViolationMetric does not enforce strict mode correctly.
Current Behavior
When strict_mode=True, the threshold is initialized as:
self.threshold = 0 if strict_mode else threshold
The success check later evaluates:
self.success = self.score >= self.threshold
Since the minimum score is 0.0, a failing role violation still satisfies:
making strict mode always succeed.
Expected Behavior
Strict mode should fail whenever any role violation is detected.
Reproduction
- Create a
RoleViolationMetric with strict_mode=True.
- Evaluate a response containing a clear role violation.
- Observe that
success=True.
Proposed Fix
self.threshold = 1 if strict_mode else threshold
Files
deepeval/metrics/role_violation/role_violation.py
Summary
RoleViolationMetricdoes not enforce strict mode correctly.Current Behavior
When
strict_mode=True, the threshold is initialized as:The success check later evaluates:
Since the minimum score is
0.0, a failing role violation still satisfies:making strict mode always succeed.
Expected Behavior
Strict mode should fail whenever any role violation is detected.
Reproduction
RoleViolationMetricwithstrict_mode=True.success=True.Proposed Fix
Files