New Mexico authorities, however, said they had yet to identify the remains, discovered Monday. And prosecutors said they were awaiting word on the cause of death before deciding on any charges.
The boy, Abdul-ghani Wahhaj, would have turned 4 Monday. Prosecutors said he was snatched from his mother in December in Jonesboro, Georgia, near Atlanta.
The search for him led authorities to New Mexico, where 11 hungry children and a youngster's remains were found in recent days at a filthy compound shielded by old tires, wooden pallets and an earthen wall studded with broken glass.
The missing boy's grandfather, Siraj Wahhaj, a Muslim cleric who leads a well-known New York City mosque, told reporters he had learned from other family members that the remains were his grandson's.
The imam said he did not know the cause of death.
"Whoever is responsible, then that person should be held accountable," Wahhaj said.
In an interview with WSB-TV in Atlanta, the boy's mother also called for "justice" as she described how her life had been taken from her after her son was abducted by his father, which she said was out of character for him. She and Siraj Ibn Wahhaj, the imam's son, had been married almost 14 years.
"I wasn't able to save my son," she told the television station.
In Facebook messages, Naeemah Rashid, Ramzi's sister-in-law, told The Associated Press that she, too, was surprised by Wahhaj's actions, saying he had always valued the closeness of their family.
Rashid also recalled from Atlanta the deep bond between the boy and his mother, who couldn't leave the room without him crying.
While the boy could not walk, Rashid remembers that he smiled and laughed as he watched his cousins play.
"Of course this is a hard time for her," Rashid said of the mother.
Ramzi, who is from Morocco, filed for divorce in December - the same month neighbors say Siraj Ibn Wahhaj and others arrived in Amalia, New Mexico.
A Georgia arrest warrant accused him of kidnapping his child. Authorities said the father at some point told his wife he wanted to perform an exorcism on the boy, who suffers seizures and requires constant attention because of a lack of oxygen and blood flow at birth.
The child's father was among five adults arrested on suspicion of child abuse in the raid at the compound. In court papers, prosecutors also said Wahhaj had been training children there to carry out school shootings.
Speaking at his Brooklyn mosque, the elder Wahhaj said he had no knowledge of any such training.
"It sounds to me it sounds crazy. But I don't know," he said. "I make no judgments yet because we don't know."
The imam's mosque has attracted a number of radicals over the years, including a man who later helped bomb the World Trade Center in 1993.
In a video posted Thursday on Facebook, mosque spokesman Ali Abdul-Karim Judan called the case a "domestic situation" and vehemently denied it had anything to do with extremism.
"None of the charges had anything to do with anybody teaching anybody shooting to commit acts of terrorism or to go in and shoot up any school," he said. "Because it's a Muslim and the circumstances that are surrounding their situation, they want to change the narrative."
The elder Wahhaj said all 11 of the children, ages 1 to 15, were either his biological grandchildren or members of his family through marriage.
"I'm very concerned with the condition of my grandchildren," he said.
He said he didn't understand why his son had taken the family and disappeared into the desert, but suggested a psychiatric disorder was to blame.
"My son can be maybe a little bit extreme," he said, though he added that he never thought he was extreme enough to kill anyone. "High-strung," he said.
The elder Wahhaj said he did not know anything about his son wanting to perform an exorcism on the boy. But he said his son and one of his daughters had become "overly concerned" with the idea of people becoming possessed.
New Mexico's Office of the Medical Investigator said it was still working to identify the remains.
Dr. Kurt Nolte, New Mexico's chief medical investigator, said the remains "are in a state of decomposition that has made identification challenging." The agency said the process could take weeks.
Taos-area District Attorney Donald Gallegos said he will await the findings on how the boy died.
Sheriff Jerry Hogrefe said the FBI put the place under surveillance in recent months that included photographs of the compound and interviews.
He said the images were shared with the mother of Abdul-ghani, but she did not spot her son, and the photographs never indicated the father was at the compound, leaving the sheriff without the information he needed to obtain a search warrant.
That changed when Georgia authorities received word that children inside the compound were starving, Hogrefe said.
The elder Wahhaj said the tip came to law enforcement through him. He said he was able to learn their whereabouts from a note that his daughter, one of the five adults at the site, sent to a man in Atlanta saying they were starving and asking for food.
That man then notified Wahhaj, who said he decided to send food and contact police.
Groves reported from New York. Associated Press writers Brinley Hineman in Atlanta and Mary Hudetz in Albuquerque, N.M., contributed to this report.
Copyright 2018 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.