KEMBAR78
Rewrite cpio pkg tests as fuzzy tests by RiSKeD · Pull Request #2528 · u-root/u-root · GitHub
Skip to content

Conversation

@RiSKeD
Copy link
Contributor

@RiSKeD RiSKeD commented Oct 17, 2022

Commit 20f840e:
I rewrote some of the cpio testing as fuzzing tests, added new ones, and fixed #935.
The baseline coverage when running go test will remain the same, because we are using the same inputs as before as seed inputs for the fuzzing tests. Additionally if specified using go test-fuzz=FuzzXXX any of the tests can be run according to the Go Fuzzing Docs as a fuzzing test as well.

Commit 9f57b69:
Provided a failing input which results in a panic for one of the tests when reading a record of a specific format.
I will file a new issue to get it fixed.

I would like to propose extending the CI environment to incorporate these new fuzzing tests in the next step.
OSS-Fuzz offers a good solution for continuously fuzzing entirely separated from GitHub and CircleCI.

@RiSKeD
Copy link
Contributor Author

RiSKeD commented Oct 20, 2022

Because of the provisioned input in commit 9f57b69 one of the fuzzing tests will fail until #2529 gets fixed.

More context: With this fuzzing approach, I also wanted to test for inconsistencies when doing multiple parsing rounds (multiple read-writes). Parsing an object, writing it back, and parsing it again should result in the same struct that was parsed in the first round. By doing this we can also ensure that no unexpected panics are occurring either.

@RiSKeD RiSKeD added the Awaiting reviewer Waiting for a reviewer. label Oct 24, 2022
@RiSKeD RiSKeD force-pushed the cpio-fuzz branch 3 times, most recently from 8f613f8 to cf90b2c Compare October 25, 2022 13:07
@RiSKeD
Copy link
Contributor Author

RiSKeD commented Oct 25, 2022

So i simplified and condensed the fuzzing for the newc format into a singular test. What it does:
It tries to parse any generated fuzzing input. If an error occurs, which is not unexpected as even unparseable inputs are generated, the test will just skip to the next input. If an input is parseable though the test will make sure it can be written back to an byte array and parsed back again. The reparsed object then must be identical to the first one. This makes sure that both directions (parsing and write-back) do not alter the underlying object.
Additionally any panics will also be caught.

@RiSKeD
Copy link
Contributor Author

RiSKeD commented Oct 25, 2022

Added a second commits which adds more checks when parsing cpio records.
I ran just short of 4 million different executions in about 30 minutes locally without encountering more errors.
The fuzzing is reproducible by running the following in the pkgs directory:

GOMAXPROCS=4 GOMEMLIMIT=4GiB go test -fuzz=FuzzReadWriteNewc

@RiSKeD RiSKeD requested a review from rminnich October 25, 2022 15:06
@codecov
Copy link

codecov bot commented Oct 25, 2022

Codecov Report

Base: 73.86% // Head: 73.88% // Increases project coverage by +0.01% 🎉

Coverage data is based on head (f697078) compared to base (137076d).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2528      +/-   ##
==========================================
+ Coverage   73.86%   73.88%   +0.01%     
==========================================
  Files         403      403              
  Lines       40977    40979       +2     
==========================================
+ Hits        30269    30278       +9     
+ Misses      10708    10701       -7     
Impacted Files Coverage Δ
pkg/cpio/newc.go 86.89% <100.00%> (+0.88%) ⬆️
pkg/cpio/utils.go 82.83% <0.00%> (+1.49%) ⬆️
pkg/cpio/archive.go 66.66% <0.00%> (+8.88%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@rminnich
Copy link
Member

quick question. We don't want only fuzzy tests, I assume? IOW there is value in the regular tests, since they run quickly and catch simple errors, and then fuzzing tests as well?

@RiSKeD
Copy link
Contributor Author

RiSKeD commented Oct 26, 2022

I would see fuzzing as an addition to the unit tests. But you are right, in newc_test.go one of the the unit tests is very close to the fuzzing tests. But only doing fuzzing would come with some disadvantages. You can use the unit test testvalues as fuzzing seed inputs (which are executed every time), but debugging a fuzzing test can be finicky due to the nondeterministic selection of inputs. And you only want to fuzz in some cases. This was not clear in my previous commit in which rewrote every test to fuzz.

For example: If we have a fuzzing tests (which parses, writes-back, parses, ...) the only "randomly" generated input is the parsed byte-array and not the struct object itself, therefore focusing on a realworld-cases.

Signed-off-by: Fabian Wienand <fabian.wienand@9elements.com>
Signed-off-by: Fabian Wienand <fabian.wienand@9elements.com>
@RiSKeD
Copy link
Contributor Author

RiSKeD commented Oct 27, 2022

I added the fuzzing inputs, which caused panics in the parsing function, as seed inputs. This does two things:

  1. These are run everytime we executing go test in the CI
  2. Coverage is improved as the additional parsing checks i included are now hit as well

This is pretty nice to get the baseline coverage of the tests up.

Signed-off-by: Fabian Wienand <fabian.wienand@9elements.com>
@rminnich rminnich merged commit 8016b9a into u-root:main Oct 28, 2022
Navidem pushed a commit to google/oss-fuzz that referenced this pull request Nov 7, 2022
Following the addition of some fuzzing tests
([#1](u-root/u-root#2528),
[#2](u-root/u-root#2535),
[#3](u-root/u-root#2536)) in the u-root project
and [preparations for this
integration](u-root/u-root#2543).
You can check out its [website](https://u-root.org/) for more info on
the project.

Signed-off-by: Fabian Wienand <fabian.wienand@9elements.com>

Signed-off-by: Fabian Wienand <fabian.wienand@9elements.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Awaiting reviewer Waiting for a reviewer.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reenable cpio test

2 participants