If you look at his position before the strike, you'll see that he's in a defensive position, slitghtly bent legs and arms on the side of the body - expecting a shot or a pass.
When the French shoots, the ball goes to Abel Xaviers's hand. In my opinion he didn't have time to take his hand out of the way. He tried to tackle the ball, and if you see other football players tackle, I think you'll hardly see them doing it with their arms behind their back.
The linesman exagerated when considering it hand on the ball, and not the other way around. He could only call for that if Xavier intentionally touched the ball.
Would he do it if it happened in France's box? I dont know. All I know is that he later said he was unable to sleep until 5:00 am, although he said he wasn't sorry.
That's my opinion...