It's not so much the size of the swell as the direction. The swell generally comes from the W - SW especially through the summer when most people visit the island but in the winter (following storms) it can be more of a NW direction which allows it to wrap around and hit the northside breaks. Starkbay, Northpoint,Transits and believe it or not even the Basin came get some nice barrelling waves. Unless you live on the island or spend a lot of time there in the winter months you don't ever see it. Often when it's cranking on the northside there is no private boats over there and even the ferrys cancel and it never hangs around long so it's almost impossible to score it when it's on.
I wish I had taken more photo's but I was either working or surfing when it was on, but here is one that I snapped on the Northside. I had surfed it in the morning but by the time I arrived back with my camera it had dropped considerably, if you look closely someone has just taken off.