Unless you are using pomace .... a highly questionable biproduct of the olive oil industry then it would be impossible to reach the high enough temperature to adequately crisp the chips without reaching the smoke point of olive oil. Extra Virgin Olive Oil has a smoke point of190 degrees C whereas sunflower oil smokes at 230 degrees c plus. The better the olive oil, the lower the flash point.
So you are either using a very questionable oil or are having nice flavoured but soggy chips.
Meant to say I do chip shop fish in batter fried in Olive Oil and it always turns out crispy without the oil reaching smoking point.
Never had a problem with the chips going crispy either although that might be down to the excellent quality of potatoes that are available here.
The Olive Oil used is first cold pressed Extra Virgin from Mendoza.