Previous versions of Internet Explorer had some troublesome issues with their old filter styles. Fortunately, the need for them has decreased since modern IE versions no longer rely on them.
While it is possible to utilize these filters for effects like rotation and gradients in older IE versions, it's important to recognize that they are ActiveX controls and not an inherent part of the browser itself.
This results in certain limitations and peculiarities that cannot be easily resolved. Specifically, complications may arise when combining these filters with other browser functionalities that impact the element or its surrounding layout.
I haven't personally experimented with using zoom alongside a filter rotation, but it seems like a scenario where compatibility issues could arise. In such cases, accepting that it might not function as intended in older IE versions could be necessary.
Exploring alternate methods to achieve your desired outcome without relying on zoom might prove beneficial. For text elements, adjusting the font-size or resizing images while allowing the browser to handle scaling could serve as viable alternatives.
If all else fails, offering IE8 users a slightly toned-down version of your site that doesn't emphasize zooming and rotation as prominently as on other browsers may be the most practical solution.