(AFP) - Forces allied with Libya's unity government said Saturday they had recaptured the port in the jihadist stronghold of Sirte from Islamic State group fighters who are now surrounded inside the city.

The fall of Sirte, the hometown of ousted dictator Moamer Kadhafi, would be a major setback to the extremists who have also lost territory in Syria and Iraq where they have declared an Islamic "caliphate".

The Libyan forces also retook a residential area in the east of Sirte, the main IS base in the North African country, a spokesman for the forces, Rida Issa, told AFP.

The jihadists are now encircled in an area of around five square kilometres inside the city, he said.

The rapid pace of the anti-IS advance has surprised the Libyan authorities.

"The battle wasn't as difficult as we thought it would be," a Libyan government official said Friday.

Foreign intelligence services estimate the extremist group has 5,000 fighters in Libya, but its strength inside Sirte and the number of civilians living in the city are unclear.

Libya's unity government forces have fought fierce street battles with the jihadists around a sprawling Kadhafi-era conference centre which once hosted international summits but now houses an IS command centre.

An AFP correspondent at the scene reported heavy street fighting on Friday about two kilometres (one mile) from the Ouagadougou centre.

- 'We won't back down' -

GNA forces used tanks, rocket launchers and artillery, the correspondent said, while the jihadists responded with machineguns, mortar rounds and sniper fire.

"We are fighting between houses, on the streets, and we won't back down before we eliminate them," said one GNA combatant, who declined to be named.

Warplanes have carried out air strikes around the conference centre and other IS positions inside the city, according to social media accounts belonging to the anti-jihadist operation.

Eleven members of the forces loyal to the UN-backed Government of National Accord (GNA) were killed and 45 wounded on Friday, mostly by sniper fire, Issa said.

Formed under a UN-backed power-sharing deal agreed by some Libyan lawmakers in December, the GNA has been working to assert its authority but has yet to receive the official endorsement of the country's recognised parliament.

The pro-GNA forces are mostly made up of militias from western cities, notably Misrata, and the guards of oil installations that IS has repeatedly tried to seize.

The forces said on Thursday they expected to announce the liberation of Sirte in "two or three days," after thrusting into the city centre.

"We're encouraged by the progress they're making," said US special envoy Brett McGurk.

"Once you have a credible force on the ground that moves against them (IS), there is a chance that they could crack pretty quickly."

GNA head Fayez al-Sarraj called Friday on "all military forces to unite in the face of our common enemy... and to join the victorious forces".

But Ahmed al-Mesmari, a spokesman for forces loyal to a rival government based in the country's far east, described the GNA forces as "illegitimate militias, loyal to an illegitimate government".

© 1994-2016 Agence France-Presse