If you choose to walk to Trywork Point, I recommend one starting point could be the pathway down from Vaughan Court (which runs off Oceana Drive to the right), and turn left at the bottom onto a grassy walking track. If my experience is a guide, you are likely to meet happy dogs and their owners enjoying a stroll along this route.
If you choose this route, I think you should allow at least 2 hours for a one way journey that has nil or minimal walking on the cattle tracks on private land.
This walk is not for everyone.
It requires specific equipment (supported walking boots) and a reasonable level of fitness, a tallish size and common sense. There are a number of dangers to be considered; the chance of rolling or spraining an ankle on the rolling rocks, the chance of injury on the slippery slopes where the needles from the Casuarina trees form a moving mat on the ground, and the surprising number of pieces of rusty fencing wire that pop up unexpectedly. In addition, if the herd of cattle was in the vicinity where you might be trying to edge along a cattle track, there might be some associated dangers. But above all, you are skirting around private land and that needs serious consideration. If you are not very tall, you may find some of the rock climbing to be unsafe and perhaps impossible.
This walk is for the few.