My nikah has been done on phone with the man who...

Praise be to Allah, and prayers on the messenger of Allah, his family, his companions and those who were loyal to him. If he married you without your guardian “your father” or two witnesses, then the marriage is invalid. The matter is not related to the husband’s family, but rather to the wife’s. As The Messenger of Allah said: "Whichever woman married without the permission of her Wali her marriage is invalid, her marriage is invalid, her marriage is invalid. If he entered into her, then the dowry is for her in lieu of what he enjoyed from her private part. If they disagree, then the Sultan is the Wali for one who has no Wali."

So you both shall go to the Sharia court to let them decide on this issue, because divorce issues are not definitive unless with a written Fatwa.