This is due to the Magic Mouse’s built-in inertia. The scrolling action continues for a little while after you stop actively moving the scroll wheel. The result is, if you move the cursor over an on-screen control, start to scroll, then move the cursor over a different control, the newly selected control will automatically start changing.
You can avoid this by not taking your finger off the scrolling area (forcing the scroll gesture to halt immediately), or by turning off your Magic Mouse’s inertia option.
The inertia can be turned off in the Universal Access section of Mac OS X’s System Preferences. You can find instructions on how to disable it here.