COLUMBIA, S.C. (AP) — The Sons of Confederate Veterans knew an attack on their cherished symbols would come after seeing photos of the white man accused of killing nine black churchgoers displaying Confederate battle flags. They just didn’t expect it from their right flank.

Leland Summers, the group’s leader in South Carolina, now feels betrayed by the same Republicans who courted their loyalties for decades to win the conservative white votes that relegated Democrats to a permanent minority party across the South.

Last week’s shootings at a historic African-American church in Charleston were so heinous that Confederate symbols suddenly became a drag on the national aspirations of a party hoping to regain the presidency in 2016. Shocking photographs of the suspect displaying Confederate symbols made it easy for Republicans to break away.

South Carolina Gov. Nikki Haley spent an intense weekend consulting with party leaders before urging her state’s lawmakers to remove the rebel banner from the Statehouse grounds. Alabama Gov. Robert Bentley then acted by executive order, taking down Civil-War era flags outside the Montgomery capitol, where the Confederacy was born.

“We knew it was coming,” Summers said Thursday, but these moves by conservative governors came as complete surprise.

Just how much influence the Sons of Confederate Veterans and similar organizations still have may become apparent in Mississippi’s elections this year.

Several powerful Mississippi Republicans called for the Confederate symbol to be removed from the state’s flag, even though voters overwhelmingly endorsed the current design in 2001. Gov. Phil Bryant has not declared his intentions yet, but the entire legislature and all statewide officers will be pushed to take their stands.

The Sons don’t plan to surrender.

In South Carolina, where the group helped defeat Gov. David Beasley in 1998 after he suggested moving the rebel flag from the Statehouse dome, they are vowing to fight any effort to send it off to the state’s Confederate Relic Room or some other museum.

In Alabama, they are already taking aim at Bentley.

“He’s bowing to political pressures without taking into consideration the blowback that’s going to come to him through that,” said Mike Williams, a group leader. “This is the state of Alabama. We do have pride.”

But contrary forces have been massing quietly for a while across the region.

Much of the South’s new population has migrated from elsewhere, and lack ties to its past. Census data shows native-born residents declined in every state of the Confederacy from 2000 to 2010, led by South Carolina, which dropped from 64 percent to 59 percent native-born.

Auto and airline manufacturing and other new industries brought in new crops of employees and executives unwilling to stomach unfavorable publicity in an era when social media can spread bad news in an instant.

The Sons of Confederate Veterans were founded in 1896 as a way to honor ancestors who fought for the South, placing historically accurate Confederate flags on Southern soldiers’ graves and re-enacting Civil War battles.

By the 1950s and 60s, the symbols took on new meanings, as segregationist politicians hoisted flags to protest federal efforts to establish equal rights for all races. A generation later, as civil rights activists demanded that the flags come down, Summers and others insisted that these symbols are about heritage, not racism. And Republicans went along.

In an instant, the images of Dylann Storm Roof posing with rebel battle flags changed this dynamic, said Chip Lake, a Republican strategist in Georgia.

“It’s not fair to brand all persons who support flying the Confederate flag as racist. But it’s fair to say that they’ve lost the branding war on whatever statement they want to make, whether it’s ‘heritage’ or ‘history’ or something more cynical,” Lake said.

The Southern Poverty Law Center doesn’t consider the Sons of Confederate Veterans to be a hate group, unlike the Council of Conservatives Citizens. After that group was credited for inspiring the attack on a website created in Roof’s name, Republican politicians moved swiftly this week to return thousands of dollars in donations its leader had given them.

Heidi Beirich, who follows white supremacy groups for the center, said the Sons of Confederate Veterans did have a schism several years ago between members who wanted to promote white superiority and those who didn’t want to appear racist.

Membership numbers dropped thereafter, and they haven’t had a significant flag rally in years. While thousands attended a Statehouse Confederate flag rally in 2000, only 30 or so attended Thursday’s Sons of Confederate Veterans rally in the same spot.

“The definitely aren’t the political force they used to be,” Beirich said.

The South Carolina division even saw its most powerful figure surrender Thursday.

Glenn McConnell, a longtime member, spent 14 years as Senate President Pro Tem, one of the state’s most influential elected offices. He brokered the 2000 compromise that brought the flag from atop the dome to a monument out front, and said it would never move again.

McConnell, now president of the College of Charleston, said even he supports his governor’s call to remove the flag altogether.

“Today is a different time,” McConnell wrote in a statement. “In the aftermath of the horrific tragedy that spilled the blood of nine souls within the hallowed halls of Mother Emanuel A.M.E. Church, the time has come to revisit the issue of the Confederate soldier’s flag, which a number of our citizens regard as offensive.”

One of the few who did show up at Thursday’s rally was Dickie Phalen from Reevesville, who had seven ancestors fight in the Civil War. Six died.

“We’ve always been outnumbered. We’ve always been outgunned,” Phalen said. “But we’re still fighting.”