KEMBAR78
Make `Lifecycle.PER_EXECUTION` more realistic by fmeum · Pull Request #867 · CodeIntelligenceTesting/jazzer · GitHub
Skip to content

Conversation

@fmeum
Copy link
Contributor

@fmeum fmeum commented Oct 16, 2023

No description provided.

@fmeum fmeum requested a review from a team October 16, 2023 08:19
@fmeum
Copy link
Contributor Author

fmeum commented Oct 16, 2023

BatikTranscoderFuzzer emits a warning on macOS that I will either get rid of or ignore.

@fmeum fmeum force-pushed the FUZZ-793-test-instance branch 3 times, most recently from eb20d6e to 0f1d72a Compare October 16, 2023 09:46
@fmeum
Copy link
Contributor Author

fmeum commented Oct 16, 2023

Stacked onto #868

@fmeum fmeum force-pushed the FUZZ-793-test-instance branch 2 times, most recently from 1794ad3 to a0982aa Compare October 16, 2023 12:54
Copy link
Contributor

@bertschneider bertschneider left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Puh, what a PR! Great additions and simplifications, thx.
Just some minor remarks, feel free to merge.

@fmeum fmeum force-pushed the FUZZ-793-test-instance branch from a0982aa to e907dff Compare October 18, 2023 07:48
fmeum added 9 commits October 18, 2023 09:48
If a target doesn't expect a finding, it doesn't have to disable this
check manually.
We also track the instance on which a given method is invoked and verify
that the `ExtensionContext` contains consistent information. This
prepares for future changes to `Lifecycle.PER_EXECUTION`.
Previously, `dumpReproducer` was called for `@FuzzTest`s using Autofuzz,
which is implemented as a static fuzz target method.
Each execution uses its own dedicated test class instance and also runs
preprocessors.

See the comment in `JUnitLifecycleMethodsInvoker#beforeFirstExecution`
for an explanation of how this still falls short of emulating default
JUnit behavior.
The check generates a warning for essentially all our tests and doesn't
seem to be easy to make more precise.
Previously, IDE executions of JUnit fuzz tests registered a
`findingHandler` in `FuzzTargetRunner` whereas CLI executions did not.
This lead to inconsistent behavior that was hard to reason about and a
lack of feature parity between the two modes (e.g. `--keep_going` was
only supported on the CLI). 

Instead, we now use a `findingHandler` to report the last, "fatal",
finding in structured form to `FuzzTestExecutor`, with all other
findings having their stack traces printed. `JUnitRunner` now handles
findings from lifecycle methods correctly, including for the exit code.
@fmeum fmeum force-pushed the FUZZ-793-test-instance branch from e907dff to 03ba60d Compare October 18, 2023 07:48
@fmeum fmeum force-pushed the FUZZ-793-test-instance branch from 03ba60d to df43aae Compare October 18, 2023 07:58
@fmeum fmeum requested a review from bertschneider October 18, 2023 08:11
@fmeum fmeum merged commit 13a7c77 into main Oct 18, 2023
@fmeum fmeum deleted the FUZZ-793-test-instance branch October 18, 2023 09:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants