On Monday, the largest group yet of such refugees, nearly 2 000, was transported to villages and the town of Konduga. Though they are just 35 km (22 miles) from Maiduguri, northeast Nigeria's biggest city, they are also on the fringes of the Sambisa Forest where the Islamic extremists still have strongholds.
How this group of returnees survives, and whether the military can protect them, could influence other refugees whom the government is keen to resettle. Maiduguri alone is home to about 1 million of the 2.6 million people forced from their homes in Nigeria and neighboring countries during Boko Haram's seven-year uprising that has killed some 20 000.
Despite the threat from insurgents, "food is the most important issue," said one returnee, Baari Mustapha. "If not, there will be serious hunger and starvation."
Food is already a critical issue. Children are dying of starvation in refugee camps like Dalori, where many of the returnees had been living in tattered tents and makeshift huts of straw on the outskirts of Maiduguri.
Dalori residents were among hundreds who protested last week, accusing officials of stealing their food donations. Nigeria's government is investigating the charges.
The refugees say the camps are miserable, and they hope life will be better back home.
"Life has not been easy in the camp," said Bintu Ganaye, a 32-year-old mother surrounded by her five children, ages 3 to 13. "Our major fear is that we don't know what the future holds out for us as we return empty-handed."
Almost all the returnees are children, women and old men. Most women have no idea of the fate of husbands and sons, who may have been killed by Boko Haram or kidnapped to be turned into fighters.
The Borno state government some months ago tried to force refugees to return home, only to meet resistance from people who said it wasn't safe. Smaller groups have voluntarily returned.
On Monday, the refugees waited hours in sweltering heat for the arrival of Borno state Gov. Kashim Shettima, who assured them that food rations would be shipped to them, along with seeds to cultivate a total of 50 hectares (124 acres) of land cleared for them.
The villagers said they believe they still could be attacked by Boko Haram on their traditional farming lands, which are located away from their villages.
Hundreds of thousands of peasant farmers have been driven from their lands by Boko Haram.
Konduga town was attacked many times in the past because it serves as a final defense for Maiduguri, the birthplace of the extremist group and headquarters of the military campaign to curb its insurgency.
Even in Dalori camp, the refugees were not safe. In January, Boko Haram sent gunmen and suicide bombers who firebombed huts there and killed at least 86 people, including children.
There has been no major attack on Maiduguri or Konduga in the months since the military and forces from neighboring countries announced they had forced the militants out of most areas except the extreme northern areas around the Lake Chad Basin, where Nigeria's borders meet Cameroon, Chad and Niger, and the Sambisa Forest to the west of Konduga.