Thanks for the pr.
Two things:
a) did you benchmark it? I would guess probabilistic use of tokes makes sense for a large number of elements
b) maybe make this an env variable instead of a commandline switch? Sth like AFL_MAX_DET_EXRAS

I thought about this and I propose something different. the extras dictionary grows to as many entries as which are to be added, no limit. IMHO it doesnt make sense to have a limit.
remember that there is not only -x file, but also -x directory (adding all files in that directory), plus the LTO autodictionary feature.
and checking before how many entries that would be and then change config.h or using -g is cumbersome.
if there are entries that the user wants to have added - they should be added.

As far as I understand (@antonio-morales correct me if I'm wrong), you can add an infinite amount of dict entries, but if you pass this threshold of currently 200, they are sometimes skipped, probabilistically
The important lines are:

      /* Skip extras probabilistically if afl->extras_cnt > MAX_DET_EXTRAS. Also
         skip them if there's no room to insert the payload, if the token
         is redundant, or if its entire span has no bytes set in the effector
         map. */

      if ((afl->extras_cnt > MAX_DET_EXTRAS &&
           rand_below(afl, afl->extras_cnt) >= MAX_DET_EXTRAS) ||
...

This seems like an optimization so other, non-dict things also get a chance.
Making the threshold configurable via env makes sense, but you may actually not change it too much, even if you have more tokens

@domenukk @vanhauser-thc Yes, I have used bigger dictionaries a few occasions in the past (modifying MAX_DET_EXTRAS constant and rebuilding the code). I haven't' benchmarked it but, if I'm not mistaken, it's a linear increase in time or O(n).
For example, if we have a file of size 512 bytes:

• 200 dictionary entries => 512 x 200 entries x 2 (over & insert) = 204800 iterations aprox
• 400 dictionary entries => 512 x 400 entries x 2 (over & insert) = 409,600 iterations approx

I think that dictionaries with more than 200 entries are useful, for example, if you're fuzzing complex interpreters/compilers.

The variable AFL_MAX_DET_EXRAS looks like a good idea to me. And it allows you to save "-g" flag for a future feature 😄

And I've kept the probabilistic functionality because I think that it can be interesting in certain cases or scenarios.
There are 2 examples:

• You have created a dictionary with 400 entries, and you want to use all of them. So you can set AFL_MAX_DET_EXRAS=500, and AFL will use all the tokens.
• You use an automatic tool for creating dictionaries, and this tool has generated a dictionary with 5000 entries. However, you think not worth using all of these entries in all the iterations. So you set AFL_MAX_DET_EXRAS=1000, and AFL will use the entries in a probabilistic way.

Can you take a look if 1301552 is a good solution to your problem?

@domenukk this means the user has to make an extra effort in first needing to check how many dictionary entries there are to be loaded. With lto autodict this is additionally cumbersome. We should simply load all supplied. Expect the user to know what he/she is doing.

Ah you mean like a AFL_EXTRAS_DET_ALWAYS?

Just as a clarification, the important loop here:

no I mean currently there is a limit on entries that afl-fuzz reads in - and aborts otherwise.
try afl-fuzz -x dictionaries/ -i in -o out ./test-instr

Ah @vanhauser-thc gotcha. How about #523 ?

If #523 is merged is this issue/feature done or is there anything still open?

@domenukk @vanhauser-thc I think 1301552 and #523 work well, so this issue can be closed now 👍