You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: Docs/Discovery.md
+6-4Lines changed: 6 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
## Introduction
4
4
5
-
One of the most important jobs of the **Test Adapter for Catch2** is to discover Catch2 test cases. Of course the [default discovery mechanisms](#default-discovery-mechanisms) that come out of the box with Catch2 are supported. However, there is also support for a [custom discovery mechanism](#custom-testcase-discovery).
5
+
One of the most important jobs of the **Test Adapter for Catch2** is to discover Catch2 test cases. Of course the [default discovery mechanisms](#default-discovery-mechanisms) that come out of the box with Catch2 are supported. However, there is also support for a [custom discovery mechanism](#custom-testcase-discovery). Note, that use of duplicate testcase names results in discovery failure, and an appropriate warning will be logged. In Catch2 v3 it is possible to have identical testcase names as long as the tags are different. However, the test explorer in Visual Studio is not able to deal with this. In this case the name of the duplicate testcase names will be modified (_e.g._, prepended with `[[DUPLICATE 1>>`), so it is clear which testcases have this problem. When those tests are run they will be skipped/fail.
6
6
7
7
## The discovery process
8
8
@@ -24,13 +24,15 @@ This is a very important step. Typically the **Test Adapter for Catch2** is prov
24
24
The **Test Adapter for Catch2** supports three different discovery mechanisms:
25
25
26
26
-`--list-tests` based (default setting)
27
-
-`--list-test-names-only` based
27
+
-`--list-test-names-only` based (not recommended, and removed in Catch2 v3)
28
28
- custom (xml based)
29
29
30
30
As of version 1.5.0 of the **Test Adapter for Catch2**, support for the Catch2 discovery command line options in combination with the `--verbosity high` option was added. Adding the latter option allows discovery of information about the source file and line number the testcase can be found at. The `--verbosity high` option was added to the default setting as of version 1.5.0 of the **Test Adapter for Catch2** as it is supported by Catch2 version 2.0.1 and up.
31
31
32
32
Before version 1.5.0 the default was not able to retrieve information about the source file and line number the testcase could be found at. This is basically the reason that the **Test Adapter for Catch2** has support for a [custom discovery mechanism](#custom-testcase-discovery), which is explained below.
33
33
34
+
As of version 1.8.0 of the **Test Adapter for Catch2**, support for the Catch2 v3 capability to set a reporter for discovery is supported. More specifically the output generated by the `xml` reporter in combination with `--list-tests`. This effectively removes the need for the custom discovery option, when Catch2 v3 is used.
35
+
34
36
As discovery of testcases requires the provided executable to be run, there is a chance that for whatever reason the executable does not stop automatically after it is run. Maybe it accidentally passed the filename filter in [step 1](#step1-filename-based-filter), and it is actually a process that runs forever. For this reason you can configure a [`<DiscoverTimeout>`](Settings.md#discovertimeout) that will kill the process if the discovery process takes longer than the set timeout. By default, this timeout is set to 1 second, which is typically more than enough.
35
37
36
38
### Step3: Extract testcases from output
@@ -49,11 +51,11 @@ This mechanism is used by default. It can discover both testcase names and tag n
49
51
50
52
### `--list-test-names-only`
51
53
52
-
This mechanism is only able to discover testcase names. It is however more robust in its ability to discover testcase names. Note however, that this discovery mechanism has been put on the Catch2 deprecation list.
54
+
This mechanism is only able to discover testcase names. It is however more robust in its ability to discover testcase names. Note however, that this discovery mechanism has been put on the Catch2 deprecation list and has been removed in Catch2 v3.
53
55
54
56
## Custom discovery mechanism
55
57
56
-
When custom discovery is used the output to be processed is expected to be Catch2 Xml (_i.e._, the same Xml used by the Catch2 xml reporter). Initially this was introduced to enable discovery of information about the source file and line number the testcase can be found at. I was unaware of the possibility to use the `--verbosity high` option to get this information and came up with this solution. As of version 1.5.0 of the **Test Adapter for Catch2** awareness was raised and the need for custom discovery is diminished. However, there may still be use cases for it, so the feature remains.
58
+
When custom discovery is used the output to be processed is expected to be Catch2 Xml (_i.e._, the same Xml used by the Catch2 xml reporter). Initially this was introduced to enable discovery of information about the source file and line number the testcase can be found at. I was unaware of the possibility to use the `--verbosity high` option to get this information and came up with this solution. As of version 1.5.0 of the **Test Adapter for Catch2** awareness was raised and the need for custom discovery is diminished. However, there may still be use cases for it, so the feature remains. Note, if you make use of Catch2 v3 there should be no need to make use of this feature.
Copy file name to clipboardExpand all lines: Docs/Settings.md
+12-6Lines changed: 12 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Settings for Test Adapter for Catch2
2
2
3
-
> The information on this page is based on **Test Adapter for Catch2** v1.7.0.
3
+
> The information on this page is based on **Test Adapter for Catch2** v1.8.0.
4
4
5
5
In order for the **Test Adapter for Catch2** to do its job, it requires certain settings to be set explicitly by the user. This is done via a _.runsettings_ file. The settings for the **Test Adapter for Catch2** are collected inside the `<Catch2Adapter>` node that can be added to the `<RunSettings>` node of the _.runsettings_ file. Below is the list of settings that are available for the **Test Adapter for Catch2**. The ones with an asterisk are required to be set by the user and have defaults that will cause the **Test Adapter for Catch2** to not discovery tests.
6
6
@@ -35,9 +35,13 @@ The following _.runsettings_ file examples only contains settings specific to th
35
35
<Catch2Adapter>
36
36
<CombinedTimeout>60000</CombinedTimeout><!-- Milliseconds; Introduced in v1.6.0 -->
37
37
<DebugBreak>on</DebugBreak>
38
-
<DiscoverCommandLine>--verbosity high --list-tests *</DiscoverCommandLine>
38
+
<DiscoverCommandLine>--verbosity high --list-tests --reporter xml *</DiscoverCommandLine>
@@ -113,11 +117,13 @@ With the `<DebugBreak>` option you can turn on or off the break on test failure
113
117
114
118
## DiscoverCommandLine
115
119
116
-
Default: "--verbosity high --list-tests *"
120
+
Default: "--verbosity high --list-tests --reporter xml *"
117
121
118
122
With the `<DiscoverCommandLine>` option you set the commandline arguments to call a Catch2 executable with in order to discover the tests that are contained within the executable. You have the choice of the test discovery options that come out of the Catch2 box (`-l`, `--list-tests`, `--list-test-names-only`) or you can provide a custom one. The only requirement for the custom discoverer is that it generates Xml output according to the Catch2 xml reporter scheme. For the build in discovery options you can add filters to select only a subset of tests. For a custom discovery option, it is up to you if you want to support test filtering on this level. For a detailed description about the discovery process see the [discovery documentation page](Discovery.md) where you can also find a custom discovery example.
119
123
120
-
> Default value changed in v1.5.0
124
+
When you use Catch2 v3, you can set the reporter to xml for improved discovery. For previous version of Catch2 the setting had no effect on the discovery output.
125
+
126
+
> Default value changed in v1.5.0, and v1.8.0
121
127
122
128
## DiscoverTimeout
123
129
@@ -151,7 +157,7 @@ With the `<ExecutionMode>` option you can choose the way tests are executed.
151
157
| ExecutionMode | Description |
152
158
|:--------------|:------------|
153
159
| Single | For each test case a separate instance of the test executable is started. |
154
-
|Combined| A single test executable is started to run multiple test cases. |
160
+
|Combine| A single test executable is started to run multiple test cases. |
0 commit comments