The coast was coated with rancid-smelling sticky white clumps of the oil as it washed in Sunday, with 11 beaches still closed to swimmers Monday in the height of a summer heatwave.
There are still lumps of the solidified oil on the beaches and the sea water in some areas is greasy.
Hong Kong comprises more than 200 islands with glittering bays, but there are increasing concerns about pollution and rubbish blighting its shores.
On Pui O beach, on the island of Lantau, cleaners raked through the famous black sand Monday morning retrieving lumps of palm oil mixed with other trash, from plastic water bottles to children's toys.
Although there is still a red flag up and the beach is officially closed, some people still ventured into the water.
One 61-year-old surfer, who gave his name as Simon and is a regular at the beach, said there was still oil in the water.
"It got under my feet and on my board. It's all slippery," he told AFP.
"Yesterday there was big chunks along the beach and in the water."
He added that there was often rubbish on the beach, often left by visitors.
"I live here now, I have to put up with it. I don't like it," said Simon, who is an airport worker originally from Hawaii.
Beach announcements told determined swimmers at the closed beaches to get out of the water Monday.
But Agnes Mercado, 49, a regular at secluded South Bay on Hong Kong Island, was determined to take her morning dip, although she said she would not submerge her upper body.
"Of course I'm worried about it, but it's even worse than this on some days," she said of the pollution.
The Leisure and Cultural Services Department said beach workers were using absorbent strips to prevent the spread of the oil and that government departments had been notified to clean up the water.
In a statement late Sunday it described the palm oil as "harmless to the human body".
But environmentalists still fear the potential impact and say the government has not done enough to contain the spillage.
"Whilst we may not see birds covered in black oil, palm oil is hazardous to wildlife in that it attracts bacteria," said Gary Stokes of the Sea Shepherd Conservation Society.
Stokes added that the oil will dissolve and break down, which could also remove oxygen from the water and kill marine life.
Swathes of rubbish frequently clog the coastline with authorities and environmentalists pointing the finger at southern mainland China as the source.
However, campaigners also say Hong Kong itself has a terrible track record on dumping of waste -- the city's landfills are groaning at capacity and there is no widespread recycling culture.