I used to sail from Dolls Point to Kurnell all day long. (When I was a newbie I did it to avoid gybing

)
Only time I've got into trouble is when the wind died, and I had to paddle all the way back. An increasing NE wind is only going to blow you back to the beach anyway. I wouldn't do it in any wind with west in it.
As long as you don't do it near sunset, hard to see you getting into any trouble, only inconvenienced, if you have equipment failure.
A little common sense in choosing the right conditions and time will go a long way to staying safe.
If you ask around on the day, you can grab a couple of guys to do the run with you. We've sometimes done it in a group of 4 or 5, which makes it safer, and a lot more fun.