The techniques used to make CPUs fast have possible vulnureable areas (not all of it is due to Out-Of-Order Execution, some was due to optimizations made on top).

The mitigations introduced stalls into those mechanism, clearing some of the state, resulting in slowdowns.

However, the mitigations are not as big slowdowns as not having the speculative execution mechanism at all.