I suspect I would have reacted rather strongly.
Such as, telling him in no uncertain terms that I expected him no to put his fingers on my stuff.
And/or call the FA right away to deal with the situation.
OTOH, maybe I look mean enough so that these things don't happen to me? :-)