As the last poster above hints, a substantial part of the problem is WinDoze. W is a multi process system, that is usually doing other things at the same time, and will be constantly distracted, and there will be delays, and there's nothing you can do about it. Like is said above, it's running battle. All windows software is constantly being 'improved' with extra bells and whisttles. Every improvement takes back some, most or MORE of the gains provided by faster machines/CPUs etc.
I admit it, I'm an old-time DOS man. I use WinDoze, in effect, under duress. I use XP because the text interface/Command prompt is more reliably DOS than anything later.
However, ALL my midi stuff is on an old DOS-only Pentium 75 sitting next to the XP machine, all my midi modules etc are attached to that, the Roland sound card is in there (more modern macnines don't even have an ISA slot for it). Midi works FINE, never any hint of any speed problems. What's 'latency' (rhetorical question !).. Midi isn't slower, it's the newer machines it's being run on that are too busy trying to do other things at the same time.
Yes, I understand, you can do some neat things via Windows. But be prepared to pay the cost, and accept it.