Terri Chung, Bae's sister, said in a statement, however, that relatives are encouraged by a growing number of people - including the Rev. Jesse Jackson - calling for her brother's freedom.
"It has been 474 days since Kenneth has been detained in the DPRK," the statement said, referencing the nation's formal name, the Democratic People's Republic of Korea. "Kenneth is just an ordinary American father of three who is desperately trying to return to his family."
The statement continued, imploring U.S. and North Korean leaders "to work together to let this U.S. citizen come home to his family."
The 45-year-old from Lynnwood, who had been living in China for seven years, was taken into custody in November 2012 while leading a tour group into a North Korean economic zone.
The cancellation, announced by the State Department on Sunday, comes only days after a detained Bae told a pro-Pyongyang newspaper that he expected to meet this month with U.S. envoy on North Korean human rights issues, Bob King.
The move signals an apparent protest of upcoming annual military drills between Washington, D.C., and Seoul, South Korea. North Korean officials also say the U.S. has mobilized nuclear-capable B-52 bombers during training near the Korean Peninsula.
North Korea calls the planned drills a rehearsal for invasion, a claim the allies deny.
The State Department said Jackson, the civil rights leader, has offered to travel to North Korea at the request of Bae's family. The State Department did not elaborate and referred questions to Jackson, whose spokesman didn't immediately respond to a request for comment.
Analysts say North Korea has previously used detained Americans as leverage in its standoff with the U.S. over its nuclear and missile programs; North Korea denies this.
The North accused Bae of smuggling in inflammatory literature and trying to establish a base for anti-government activities at a border city hotel.
Chung says Bae's Christian faith got him into trouble.
In August, North Korea also rescinded an invitation for King to visit, saying Washington had perpetrated a grave provocation by flying B-52 bombers during military drills with South Korea.
Chung said Friday that her family learned from the State Department that her brother had been taken back to a labor camp from a hospital where he had been treated after losing 50 pounds.
On Monday, Chung said in the statement that his family is "gravely concerned that the stress Kenneth endures at the labor camp will be too much for him."