Skip to content

fix(verify): default --show-builtin-errors to false#1325

Open
dgeorgievski wants to merge 1 commit intoopen-policy-agent:masterfrom
dgeorgievski:fix/show-built-in-errors-flag-default
Open

fix(verify): default --show-builtin-errors to false#1325
dgeorgievski wants to merge 1 commit intoopen-policy-agent:masterfrom
dgeorgievski:fix/show-built-in-errors-flag-default

Conversation

@dgeorgievski
Copy link
Copy Markdown

Change the default value of the --show-builtin-errors flag from true to false. This aligns the behavior of conftest verify with opa test and reduces verbosity by hiding Rego built-in errors by default.

Fixes: #1079

I have attached the original policy and tests rego files used as a demo for writing rego functions and their tests.
numbers.tgz

These tests files could be used to a demo of the change of the ``--show-builtin-errorsdefault value fromtrue` to `false`

Before

▶ conftest --version
Conftest: dev
OPA: 1.12.1

# show-builtin-erros default set to `true`
▶ conftest verify  \
--policy ./numbers \
--report=fails
Error: running verification: run test: [{eval_builtin_error to_number: strconv.ParseFloat: parsing "a": invalid syntax numbers/policy.rego:51 0x14000a09620} {eval_builtin_error to_number: strconv.ParseFloat: parsing "a": invalid syntax numbers/policy.rego:59 0x14000a40030} {eval_builtin_error to_number: strconv.ParseFloat: parsing "a": invalid syntax numbers/policy.rego:67 0x14000a40990}]

# show-builtin-erros set to `false`
▶ conftest verify  \
--policy ./numbers \
--report=fails --show-builtin-errors=false
numbers/policy_test.rego:
data.acme.numbers_test.test_compare2num_a_null_eq_b_cannot_compare: PASS (3.982ms)
data.acme.numbers_test.test_compare2num_a_str_eq_b: PASS (4.143583ms)
data.acme.numbers_test.test_compare2num_a_lt_b: PASS (4.135542ms)
data.acme.numbers_test.test_compare2num_a_letter_eq_b_cannot_compare: PASS (4.544875ms)
data.acme.numbers_test.test_not_3_is_gt_5: PASS (650.208µs)
data.acme.numbers_test.test_compare2num_a_gt_b: PASS (5.065709ms)
data.acme.numbers_test.test_compare2num_a_eq_b: PASS (1.145416ms)
data.acme.numbers_test.test_not_5_is_gt_5: PASS (5.217917ms)
data.acme.numbers_test.test_7_is_gt_5: PASS (1.174417ms)
data.acme.numbers_test.test_any_match_ok: PASS (5.327958ms)
data.acme.numbers_test.test_allow_ok: PASS (5.429625ms)
data.acme.numbers_test.test_any_match_fail: PASS (5.521375ms)
data.acme.numbers_test.test_allow_fail: PASS (6.16575ms)
--------------------------------------------------------------------------------
PASS: 13/13

After the change; Binary compiled with make from the repo

~/Projects/conftest/conftest --version
Conftest: 2f12-dirty
OPA: 1.15.2

# show-builtin-erros default set to `false`~/Projects/conftest/conftest verify  \
--policy ./numbers \
--report=fails
numbers/policy_test.rego:74:
data.acme.numbers_test.test_compare2num_a_null_eq_b_cannot_compare: PASS (3.808708ms)
numbers/policy_test.rego:58:
data.acme.numbers_test.test_compare2num_a_lt_b: PASS (3.8145ms)
numbers/policy_test.rego:62:
data.acme.numbers_test.test_compare2num_a_eq_b: PASS (3.84475ms)
numbers/policy_test.rego:66:
data.acme.numbers_test.test_compare2num_a_str_eq_b: PASS (3.912875ms)
numbers/policy_test.rego:78:
data.acme.numbers_test.test_7_is_gt_5: PASS (160.125µs)
numbers/policy_test.rego:86:
data.acme.numbers_test.test_not_5_is_gt_5: PASS (4.007834ms)
numbers/policy_test.rego:82:
data.acme.numbers_test.test_not_3_is_gt_5: PASS (159.583µs)
numbers/policy_test.rego:70:
data.acme.numbers_test.test_compare2num_a_letter_eq_b_cannot_compare: PASS (175.708µs)
numbers/policy_test.rego:54:
data.acme.numbers_test.test_compare2num_a_gt_b: PASS (4.792792ms)
numbers/policy_test.rego:18:
data.acme.numbers_test.test_any_match_fail: PASS (5.049333ms)
numbers/policy_test.rego:6:
data.acme.numbers_test.test_any_match_ok: PASS (5.091958ms)
numbers/policy_test.rego:30:
data.acme.numbers_test.test_allow_ok: PASS (5.209208ms)
numbers/policy_test.rego:42:
data.acme.numbers_test.test_allow_fail: PASS (6.88575ms)
--------------------------------------------------------------------------------
PASS: 13/13

# set `show-builtin-errors` to true~/Projects/conftest/conftest verify  \
--policy ./numbers \
--report=fails --show-builtin-errors=true
Error: running verification: run test: [{eval_builtin_error to_number: strconv.ParseFloat: parsing "a": invalid syntax numbers/policy.rego:51 0x14000b136e0} {eval_builtin_error to_number: strconv.ParseFloat: parsing "a": invalid syntax numbers/policy.rego:59 0x14000b440c0} {eval_builtin_error to_number: strconv.ParseFloat: parsing "a": invalid syntax numbers/policy.rego:67 0x14000b449f0}]

Change the default value of the `--show-builtin-errors` flag from
true to false. This aligns the behavior of `conftest verify` with
`opa test` and reduces verbosity by hiding Rego built-in errors
by default.

Fixes: open-policy-agent#1079

Signed-off-by: Dimitar Georgievski <dgeorgievski@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Set --show-builtin-errors flag to false by default

1 participant