The 14th-seeded Bears, from the same league that produced last year's tournament darlings "Dunk City" of Florida Gulf Coast, came back from five points down in the last 4:52 as Duke's offense collapsed.
Jakob Gollon scored 20 points and Daniel Coursey scored 17, helping the Bears overcome a season-high 15 3-pointers from Duke.
Mercer qualified for the tournament by winning the Atlantic Sun conference championship over Florida Gulf Coast. A year earlier, the Bears lost that game and watched FGCU advance to the Sweet 16.
Mercer scored 11 straight points during the late 20-5 run that clinched the biggest victory in school history. It's the second one-and-done in three years for the third-seeded Blue Devils.
Quinn Cook scored 23 points and Rasheed Sulaimon added 20.
But the Blue Devils' defense an uncharacteristic weakness all season did them in again while all those Mercer seniors picked Duke apart down the stretch.
Duke went up 63-58 with 4:52 left after Parker converted a three-point play and Tyler Thornton hit three free throws.
Duke didn't score again until the final minute.
Coursey countered by rattling in a jumper in the lane, and after two empty possessions for Duke, some slick ball rotation by Mercer set up an open 3 from Anthony White Jr. that tied it at 63 with 2 1/2 minutes left.
Duke star Rodney Hood picked up his fourth foul on the Bears' next possession and Gollon hit two free throws to put Mercer ahead for good.
By that point, Duke in the midst of an untimely run of empty possessions could do nothing right. Parker missed a 3-pointer in traffic before Hood was called for walking, leading White to give a fist-pump to the Mercer fans who stood all day.
The Bears hit 12 of 14 free throws in the final 2 minutes and finished 23 of 28 from the line. Gollon was 6 for 6 during that late span, and his two with 5 seconds left put the Bears up 78-68 before Cook hit a meaningless 3 with 0.2 seconds remaining.
White finished with 13 points, and A-Sun player of the year Langston Hall and Ike Nwamu added 11 apiece for Mercer which watched the last year's Sweet 16 run by "Dunk City" with a mixture of pride and envy.
The Bears not Gulf Coast were the No. 1 seed in last year's A-Sun final but were knocked off by Andy Enfield's high-flying team that went on to upset Georgetown and San Diego State in last year's tournament.
What Mercer did Friday sending home one of the sport's true blue bloods might be even more impressive because of the sheer volume of big-named talent on Duke's team.
Mercer has 1,176 wins as a program or, only 191 more than Duke coach Mike Krzyzewski has all by himself.
Parker, one of a long list of high school All-Americans on Coach K's roster, finished what might have been his final college game with 14 points. Hood a redshirt sophomore transfer from Mississippi State who also could be headed to the pros had just six points before fouling out with 1:07 left.
Parker and Hood combined to shoot 6 of 24 for Duke, which went one-and-done for the second time in three years both times on neutral courts in its home state. Two years ago, C.J. McCollum and 15th-seeded Lehigh sent the Blue Devils packing 75-70 in Greensboro.