Reputational damage has been caused to Linfield Football Club due to a "small element" of fans, according to chairman Roy McGivern.