WebTau output

[aplatypus]

• I am running tests wiht WebTau under the JUnit 5 test runner

• At the moment I have a configuration problem to resolve. JUnti 5 prints an informative report (see screenshot):
  

• I have added logging to the build script, etc but it looks like WebTau has eaten and reformatted STDOUT

  • You can see the problem in the screenshot

• Can WebTau output processing be turned off or turned down?

• The WebTau report doesn't show the STDOUT or STDERR at all. In a test environment that would be essential reading on some days.

[aplatypus]

• One thing to update. I was able to see what's going wrong by commenting-out the @WebTau annotation
• At least this got me past the config problem.
• I also discovered that the WebTau report was hiding the build output from Gradle (not clear for Maven_???_)

[MykolaGolubyev]

Webtau shouldn't eat/capture any std output.
You can pass verbosity parameter via system property or Env Variable to turn off output of webtau (it will still be available inside report).

Reformatting you see is webtau using ANSI output to print things in colors. You can use noColor system property (or part of webtau.properties file) or use WEBTAU_NO_COLOR=true env var to remove ansi. Usually CI systems recognize the colors.

Need more info on "hiding the build output". I may be missing some pieces here.
webtau doesn't know much about build system and how you run it. Definitely doesn't know about other steps in your build system.

If it helps we can do screen sharing or sample GitHub project to have established context

[aplatypus]

• I see the shortfall in describing the question. Also there are a few 'symptoms' that can be bothersome.

1. Colours -- WebTau must be filtering all the output to do the colouring. If you look at the screen shot showing the "Standard Output" box you can see ANSI sequences as you say.

   • Thanks for pointing out the WEBTAU_NO_COLOR=true. Personally, it makes more sense for OFF to be the default and colour ON as an opt-in thing.
   • JUnit doesn't emit escape codes in the output like that. This project uses Gradle, however JUnit using Maven does not change STDOUT and STDERR
   • Plain output can be passed to grep and other tools to use with automated scan and checks on the logging output when tracking down issues.
   • When a person is not troublshooting or looking at some kind of performance analysis Test code output is almost ignored.
   • Sometimes it can be handy to have coloured ouput on the command line for sure, and ...I'd recommend making 'plain' output with no-formatting the default and keep coloured output as WEBTAU_COLOR=true . It is one of those nice to have things when you are at a console.

2. Build output ... For whatever reason JUnit packages a test run output into a few 'boxes'. Test results, stdout, stderr. for each test case (see: point 3, also).

   • This is packaged in HTML files and one looks at the test results by loading:
     • file:///project_dir_path/build/reports/tests/test/index.html
   • It may be that JUnit is consuming output and WebTau is left wtih whatever residue JUnit leaves in the output stream.
     • That bears investigation I'd say becasue one would only want to look at a single test output report not two or three.
   • I do not know if JUnit provides structure for add-on tools like WebTau to integrate with the test engine.

3. Test Cases -- WebTau is reporting the @AfterAll and '@BeforeAll` set-up routines as test cases. This was a surprise.

   • It might be useful to sometimes see 'results from the set-up functions, but I suggest that ought to be an option or annotation on the method like @WebTau.report.
   • The per test set-up methods: @BeforeEach and @AfterEach are not reported. That's a little inconsistient.
   • I see some of these items as something that can be useful sometimes, normally Test output should be a as silent as possible to allow problems to be seen and reduce noise.
   • Of course that's is only one opinion/perspective.
   • A case in point; the CI from our github automation on successful runs still generates a montain of output from integration test checks. When there's a fault you have to wade through outputs until you find were the issue surfaced.
   • Less seems to be better on this topic.
   • The way WebTau puts things into categories is very good and does/will help. In this domain for web testing seeing the HTTP and JSON structures passed back-and-fro would be good too.

[MykolaGolubyev]

Colors

webtau should not modify output You provide and wrap it in colors.
To double check that, use print statement to print "hello" and see if it is being wrapped with anything

Motivation for colors by default is to address cases where people use IDE or CLI to run their tests. I personally run more tests from IDE when I write them and seeing webtau steps and their output in colors make it easier for me to digest it.
If no colors is the default, then every time you run a test from IDE you need to go and configure IDE to automatically pass color option - not ideal developer experience.

CI tools I use like Jenkins or GitHub CI properly parse the colors. For other tools noColors option is present.

Build Output

Not sure who/what packages JUnit output.
Each webtau action is automatically wrapped with step concept. Later all the steps and tests are available as rich HTML report.
Did you have a chance to look at it?

Webtau does integrates with JUnit5 API via listeners. That's how webtau HTML report associates test entries and performed steps.

Test Cases

There could be confusion with beforeAll for a test case and the entire test suite. Webtau has one for the entire test suite for things like starting servers, running CLIs in background, setting auth, etc.

Webtau made a design decision to include steps executed in each @BeforeEach block and make them part of each test steps. It was done to simplify test failure investigation. I.e. you click on a test in the report and you see all the steps takes, including before test and after test.
BeforeAll tests within a class is an entity in itself and it useful to be able to click on it and see steps and their failures.

A case in point; the CI from our github automation on successful runs still generates a montain of output from integration test checks. When there's a fault you have to wade through outputs until you find were the issue surfaced.

This is one of the reason webtau produce HTML report so you click on it and immediately see what is wrong. You can completely disable output from webtau using -DverbosityLevel=0 or WEBTAU_VERBOSITY_LEVEL=0

HTTP and JSON

if you use http. module for REST API testing, you should be able to see request/response JSON both inside console and inside HTML report. If it is not the case, I will have to investigate.