-
Notifications
You must be signed in to change notification settings - Fork 428
Rewrite cpio pkg tests as fuzzy tests #2528
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
0aeeefa to
9f57b69
Compare
|
Because of the provisioned input in commit 9f57b69 one of the fuzzing tests will fail until #2529 gets fixed. More context: With this fuzzing approach, I also wanted to test for inconsistencies when doing multiple parsing rounds (multiple read-writes). Parsing an object, writing it back, and parsing it again should result in the same struct that was parsed in the first round. By doing this we can also ensure that no unexpected panics are occurring either. |
8f613f8 to
cf90b2c
Compare
|
So i simplified and condensed the fuzzing for the newc format into a singular test. What it does: |
|
Added a second commits which adds more checks when parsing cpio records. |
Codecov ReportBase: 73.86% // Head: 73.88% // Increases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## main #2528 +/- ##
==========================================
+ Coverage 73.86% 73.88% +0.01%
==========================================
Files 403 403
Lines 40977 40979 +2
==========================================
+ Hits 30269 30278 +9
+ Misses 10708 10701 -7
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
|
quick question. We don't want only fuzzy tests, I assume? IOW there is value in the regular tests, since they run quickly and catch simple errors, and then fuzzing tests as well? |
|
I would see fuzzing as an addition to the unit tests. But you are right, in newc_test.go one of the the unit tests is very close to the fuzzing tests. But only doing fuzzing would come with some disadvantages. You can use the unit test testvalues as fuzzing seed inputs (which are executed every time), but debugging a fuzzing test can be finicky due to the nondeterministic selection of inputs. And you only want to fuzz in some cases. This was not clear in my previous commit in which rewrote every test to fuzz. For example: If we have a fuzzing tests (which parses, writes-back, parses, ...) the only "randomly" generated input is the parsed byte-array and not the struct object itself, therefore focusing on a realworld-cases. |
Signed-off-by: Fabian Wienand <fabian.wienand@9elements.com>
Signed-off-by: Fabian Wienand <fabian.wienand@9elements.com>
|
I added the fuzzing inputs, which caused panics in the parsing function, as seed inputs. This does two things:
This is pretty nice to get the baseline coverage of the tests up. |
Signed-off-by: Fabian Wienand <fabian.wienand@9elements.com>
Following the addition of some fuzzing tests ([#1](u-root/u-root#2528), [#2](u-root/u-root#2535), [#3](u-root/u-root#2536)) in the u-root project and [preparations for this integration](u-root/u-root#2543). You can check out its [website](https://u-root.org/) for more info on the project. Signed-off-by: Fabian Wienand <fabian.wienand@9elements.com> Signed-off-by: Fabian Wienand <fabian.wienand@9elements.com>
Commit 20f840e:
I rewrote some of the cpio testing as fuzzing tests, added new ones, and fixed #935.
The baseline coverage when running
go testwill remain the same, because we are using the same inputs as before as seed inputs for the fuzzing tests. Additionally if specified usinggo test-fuzz=FuzzXXXany of the tests can be run according to the Go Fuzzing Docs as a fuzzing test as well.Commit 9f57b69:
Provided a failing input which results in a panic for one of the tests when reading a record of a specific format.
I will file a new issue to get it fixed.
I would like to propose extending the CI environment to incorporate these new fuzzing tests in the next step.
OSS-Fuzz offers a good solution for continuously fuzzing entirely separated from GitHub and CircleCI.