The problem is most likely a poor design choice. I've seen this before from several mouse manufactures. In a proper design, they'd have used a LM339 to detect changes in the photo interrupters. They'd have then designed in the proper positive feedback to keep it from being noisy right at the cross over point. In fact, this was the way they were first designed. Then someone said, "all we are doing is using the expensive LM339 to create hysteresis. Why not use a 7414/74LS14 that has built in hysteresis." I don't think anyone really thought about why this was a bad idea.
Make a plot of a sine wave and a cosine wave. Make marks at the 3/4 voltage points and then look at what the desired quadrature points look like versus the actual quadrature points are. Now take one of your sine or cosine waves and shift it just a little and see what happen to the quadrature signals.
Using the 74LS14 part reduces the margin to almost nothing to make the circuit work. Likely what you are seeing is aging of one or both of the photo interrupters. Look at the signals from these on an oscilloscope. Move the mouse slowly, but smoothly. Watch the points that the photo input device triggers up and triggers down. Try to adjust the vertical such that it triggers evenly amplitude wise up and down. Now you need to either increase the resistor in series with the output of the photo interrupter or decrease the resistance of the resistor for the LED. You need to do this such that as you move the mouse smoothly, the width of the high levels match those of the low levels.
Do be careful about decreasing the LED's resistor. You don't want to use such a high level that it blows the LED up. Calculate a safe maximum current and use an appropriate resistor to protect it.
If the LED has aged so much that you can not bring the signal to even levels, you'll most likely have to replace the photo interrupter.
Before giving up, do try cleaning the LED and Photo sensor lenses with alcohol, especially if the previous user was a smoker!!
Usually it is the LED that is failing. If you can't find a matching LED, I once cut the LED off and used a red side LED and a piece of tape to make the tiny slot.
Dwight