re-rebased on #2818

rebased off master

I also kept the perf improvements from #2899 for hex-parsing, because we're still using hex-parsing for IDs, since those are short and would be kinda nuts to migrate.

+1 when green

thanks, 2693736

COBS would be more efficient for preventing embedded nulls. Use a 16-bit version of it, if a null is a 0000.

That's interesting! I would want to see a jsperf, though, to confirm that running COBS over strings is faster in most browsers than replace()ing a single character 3 times. E.g. I wonder if, by replacing a single character with more than one characters, this maybe causes an array shift under the hood, which would actually slow things down.

to confirm that running COBS over strings is faster

Faster: no. (though comparable, I would think, depending on how string manipulation is implemented.) You can run the COBS algorithm in place; it creates a string with 1 additional byte (or word) in length.

More efficient in storage: yes. The thing about replacing unicode 0000, 0001, and 0002 is that although these are unusual string characters, they are likely to occur in binary data, so if you have data with lots of 0000, then the approach used here essentially doubles the data size.

Ah, that's true. I forgot that we're replacing a single character with two characters here. OK, it's worth giving it a shot! :)