The doctype was allowed in EPUB 2:

<!DOCTYPE ncx PUBLIC "-//NISO//DTD ncx 2005-1//EN" "http://www.daisy.org/z3986/2005/ncx-2005-1.dtd">

We don't have an exception for it in Appendix B, though, so it would still be invalid in EPUB 3 even after the loosening for SVG/MathML.

I don't know how much it was used in EPUB 2, but should we add to simplify any porting forward of content?