kitserver adds more work for CPU, GPU, and operating system (memory management, for example) - there's no question about it. On some computers it is more noticeable than on others. It's not always a question of faster CPU or more RAM or newer videocard.
As correctly mentioned earlier, there's lots of factors here: bigger textures (HD kits/balls), LOD modification, crowd fix, aspect ratio correction for widescreens - all of this affects the game engine. There's no way to completely avoid that.
We (kitserver dev.team) make every effort to minimize this effect, but we don't always succeed. That said, it would be helpful, if people who can see the diffference clearly, did some investigation, for example trying to isolate exactly which kitserver module has the most adverse effect on performance. (A fair amount of work is done by kload.dll, so be sure to test the case when everything is disabled in the kload.cfg.)
Overall, this is a useful discussion, and hopefully we can improve the performance, and get back to that "smoothness" of unmodified gameplay.