Before taking this any further I'd suggest that you should re-do your experiment using 88.2kHz rather than 96kHz. Why? Because to do sampling rate conversion between 96kHz and 44.1kHz without introducing either frequency response anomalies or aliasing is very difficult. Many of the commonly available rate converters have poor measured performance doing that conversion. The reason why it's difficult is because the smallest whole number ratio that the conversion can be factored to is very large, which incidentally, was a deliberate ploy by the makers of the DAT format (48k to 44.1k conversion) to reduce piracy. 44.1k to 88.2k is simple because it's just a factor of two.
If after this experiment the results are still the same then I guess you may have more investigating to do to figure out why this might be. On the other hand, if the difference is not as dramatic than with the 44.1k / 96k test then I'd argue that most of what you are hearing is probably due to the limitations of the sampling rate converter, whether that is frequency response or aliasing or both.
Personally, I'd say the biggest problem with doing tests like this is knowing which source is which. You need a blind test where you don't know which source came from which process. Maybe you should get someone to rename the files for you and randomise the draw so you don't know what your playing. If you do know what you're playing then you'll likely be biased towards 96kHz because that must be better (higher rate must be better)! The same issues plague Hi Fi magazines because naturally the gear that cost 10 times as much must sound better.
I'd be reticent to draw the conclusion that processing at higher rates is superior because from a mathematical standpoint, such a proposition has no basis. Mathematically it is clearly inferior, but the real problem is what is added by the 96kHz processing may be audibly pleasing even though it may be due to distortion. So if you like the sound of what it adds, I have no issue with that standpoint, but if someone starts claiming that it is purer and more accurate then I'd be reticent to agree. People like adding distortions to sound. At the risk of offending many Hi-Fi purists, the essential reason tube amps continue to be made is to add pleasing distortion. That's fine by me, but I just cringe when people start claiming that it is because tubes are somehow technically superior. Measurement cannot demonstrate that assertion so it invariably ends up as an article of faith. I'm not a believer in that faith.
Different is different, not necessarily better.