ankerl's dense map is slower than std::unordered_map?

Lately I've been running into some performance Issues with std's unordered_map, I use it mainly for hash wise lookup, e.g:

std::unordered_map<uint64_t, mpark::variant<int32_t, float>>

However from my understanding this is not fully optimized? I've tried using ankerl's dense map but that ended up giving me random stutters and sometimes was just slower than unordered_map.

What do I do here?