Curious TOC problem

[David Carlisle]

The error message means you have `\addvspace`  in horizontal mode, the
fragment you show generates no error with the standard definitions

\documentclass{book}
\usepackage{hyperref}
\begin{document}

\contentsline {chapter}{Preface}{v}{section*.1}
\contentsline {section}{\numberline {1}Definition of category}{1}{section.1}
\contentsline {subsection}{\numberline {1.1}Some constructions for
categories}{2}{subsection.1.1}

\end{document}


But presumably you have definitions setting some level of the contents
"inline" as a paragraph rather than individual display lines, followed
by a display line that is expecting to be in vertical mode, without
ending the previous paragraph.