Gun calibration can be a bit tricky on CarnEvil. It is very sensitive to brightness as well as ambient light in the room. If you are having issues calibrating, make sure the brightness is turned up as much as it can go without washing out the picture. Also, if the monitor has geometry issues or is blurry, fix these before attempting to calibrate.
I've noticed that even with a good monitor with ideal light conditions, the game can sometimes refuse to calibrate. Place the gun sight directly on the monitor glass in front of the X to get the game to pick up the calibration and let you continue. You'll notice that even though you need to be next to the monitor to get the calibration to happen, when it lets you test the gun aims true and works fine.