The industry says it is lobbying the Government hard and is worried about restrictions on foreign factory workers and designers.
Adam Mansell, Chief Executive of the UK Fashion & Textiles Association (UKFT), said there was a “sense of uncertainty”.
So far, however, the industry appears to have responded positively to the result of the EU referendum.
Figures published last year by the British Fashion Council show the sector had risen in value from £26bn in 2013 to £28bn, an increase of 8%.
Employment also jumped by nearly 100,000.
Mr Mansell said that there was still “nervousness” about what the future holds for businesses.
He told Sky News it was down to a “sense of uncertainty and not quite knowing whether we are going to have a hard Brexit or a soft Brexit”.
“All the different impacts that we could have, there’s a real sense of uncertainty,” he added.
Importing goods has become more expensive because of the fall in the pound but some smaller British businesses are thriving.
London-based Fashion Enter Ltd, a factory making garments for clients such as ASOS and M&S, has never been so busy.
Production manager Caroline Ash said its order books are full 10 weeks in advance.
“We are finding it’s actually boosting our business,” she said.
“And because we have a stitching academy on site we are training new stitchers so we are not so reliant on a European workforce.”
The company also gets most of its fabric from Leicester and, therefore, is not affected by prices hikes from China.
But, like many other manufacturers in the industry, the business does rely on foreign workers.
There are questions over what potential new immigration rules would mean for highly skilled workers from Europe working in Britain.
Adam Mansell said the UKFT and others have been lobbying the Government “very hard”.
“(We want) to make sure that any immigration system that is brought in, any points-based system, recognises that the UK should be open for all talent.
“It’s not just about your lawyers and your doctors, it’s also about the people who actually sit on the machines. We need them.”
There is also concern that the next generation of designers from abroad may be held back by stricter immigration policies.
Jewellery designer Lauren Cooke, who is from Austria, said: “My concern is that I think certain taxes would be put into place, so if I sold my jewellery line in a UK store I would suddenly have to pay a large amount of tax which I wouldn’t have to pay in Europe.
“I also worry about being able to freely travel between UK and Austria.”
But, despite a looming Brexit, there still seems to be an overriding assumption that London will remain a “jewel” in the industry.