I can't seem to track down a normative requirement that the nav document be a conforming xhtml content document.

The closest we come is this statement near the end of the restrictions section:

As a conforming XHTML content document, EPUB creators MAY include the EPUB navigation document in the spine.

As that isn't a normative requirement, you could conceivably embed nav elements in an svg and call it a navigation document, since all we have are nav restrictions.

In Packages 3.2, we used to have this requirement:

It MUST conform to the content conformance constraints for XHTML Content Documents defined in XHTML Content Documents — Content Conformance [ContentDocs32].

But that requirement appears to have been lost in translation to the new spec structure.