Hey Simon,
I would initially say no go based on the following:
I believe the current in the Shortland Bluff and outer Swan Island area would make the run difficult if not impossible.
The commercial and recreational boat traffic in the river is pretty hectic at times and the current is so strong outside slack i doubt you could paddle against it. Slack only lasts 20 = 30 mins.
The timing would have to miss the ferry timetable as well.
I will ask a mate of mine Mark who is based on Swan Island about defence getting the knickers in a knot, I think they would be ok unless you land then you would get a very quick visit.
I also think parts of Swan Bay are protected habitat and you cannot touch the bottom. I was refused a permit to do SUP lessons in Swan Bay for this reason.
May I suggest with Water Support and a good plan we could do this one though! 
If we left Queenscliff on the flood tide we could drift down past the seals at Chinmans Hat and then on to South Channel Island. I think I can still get the keys to explore the underground tunnels have a bit of a BBQ while waiting for the tide to turn then make the run back.
Would be a great summers afternoon jaunt.
Phill
