Boat migrants intercepted by Australian authorities and returned to Vietnam are facing hefty prison terms, despite their country's government previously claiming those fleeing by sea would not face retribution, a leading human rights group says.

The development casts the spotlight onto the Turnbull government's insistence on returning so-called "illegal" boat passengers despite fears over their welfare. Labor also officially supports the turn-back and take-back policy, but is internally split over the issue.

Human Rights Watch says a Vietnamese court is on Thursday due to hear the case of four people charged with "organising for others to flee abroad illegally".

If convicted, the group face up to seven years in prison, Human Rights Watch Australia director Elaine Pearson said.

She did not believe the four - Nguyen Dinh Quy, his wife Huynh Thi Kieu, Tran Thi Lua and Nguyen Minh Quyet – were seeking asylum in Australia, but were poor and "looking for a better life".

Ms Pearson said those charged arranged for 42 relatives and acquaintances, including children, to join them on the boat, but were not professional people smugglers. She said passengers paid some money for the journey to cover the cost of food, fuel and buying the boat.

Human Rights Watch says the boat left the Vietnamese city of Phan Thiet on July 1 last year. It was later intercepted by Australian officials and the passengers were taken back to Vietnam by plane.

Ms Pearson said two of those returned claimed an Australian immigration official told them they would not be arrested or imprisoned upon their return, and that the Vietnamese government would find work for them and school for their children.

The defendants say once arriving in Vietnam, Australian consular authorities also witnessed a Vietnamese official telling the group they would not be arrested or detained.

However Human Rights Watch says some of the defendants were later detained without charge and denied access to a lawyer, and eventually all four were charged.

The organisation says prisoners in Vietnam endure notoriously poor conditions and inadequate medical care, and some of the defendants are suffering bad health.

She called on Vietnamese authorities to drop the charges and urged the Turnbull government to intervene, adding that "imprisoning desperate people trying to leave their own country is cruel as well as unlawful".

The incident follows the return to Vietnam of passengers from another unrelated boat in April last year.

Human Rights Watch says passengers from that boat have been charged and convicted.

This came despite Operation Sovereign Borders commander Major-General Andrew Bottrell telling a Senate hearing last May that Vietnamese officials gave a written assurance that the group would not be punished.

He admitted the Australian government did not track asylum seekers once they had been returned, and the commitment was taken on trust.

Ms Pearson said Vietnam had "blatantly broken its promise to the Australian government not to prosecute boat returnees".

Immigration Minister Peter Dutton and Labor immigration spokesman Richard Marles have been contacted for comment.

A Department of Immigration and Border Protection spokesman said "any potential criminal investigations or proceedings for people smuggling related offences are a matter for the Vietnamese Government".