Each year, the Southern Poverty Law Center, of which I am the president, compiles and publishes a census of domestic hate groups. Our list, which is cited extensively by journalists, academics and government officials alike, provides an important barometer—not the only one, of course—to help us understand the state of hate and extremism in America.

In recent weeks, we’ve received a number of requests to name Black Lives Matter a hate group, particularly in the wake of the murders of eight police officers in Dallas and Baton Rouge. Numerous conservative commentators have joined the chorus. There is even a Change.org petition calling for the hate group label.

In our view, these critics fundamentally misunderstand the nature of hate groups and the BLM movement.

Generally speaking, hate groups are, by our definition, those that vilify entire groups of people based on immutable characteristics such as race or ethnicity. Federal law takes a similar approach.

While it’s no surprise, given our country’s history, that most domestic hate groups hold white supremacist views, there are a number of black organizations on our hate group list as well.

A prime example is the New Black Panther Party (NBPP), whose leaders are known for anti-Semitic and anti-white tirades. Its late chairman, Khalid Abdul Muhammad, famously remarked: “There are no good crackers, and if you find one, kill him before he changes.” Bobby Seale, a founding member of the original Black Panther Party, has called the NBPP a “black racist hate group.”

We have heard nothing remotely comparable to the NBPP’s bigotry from the founders and most prominent leaders of the Black Lives Matter movement and nothing at all to suggest that the bulk of the demonstrators hold supremacist or black separatist views. Thousands of white people across America—indeed, people of all races—have marched in solidarity with African Americans during BLM marches, as is clear from the group’s website. The movement’s leaders also have condemned violence.

There’s no doubt that some protesters who claim the mantle of Black Lives Matter have said offensive things, like the chant “pigs in a blanket, fry ‘em like bacon” that was heard at one rally. But before we condemn the entire movement for the words of a few, we should ask ourselves whether we would also condemn the entire Republican Party for the racist words of its presumptive nominee—or for the racist rhetoric of many other politicians in the party over the course of years.

Many of its harshest critics claim that Black Lives Matter’s very name is anti-white, hence the oft-repeated rejoinder “all lives matter.” This notion misses the point entirely. Black lives matter because they have been marginalized throughout our country’s history and because white lives have always mattered more in our society. As BLM puts it, the movement stands for “the simple proposition that ‘black lives also matter.’”

The backlash to BLM, in some ways, reflects a broad sense of unease among white people who worry about the cultural changes in the country and feel they are falling behind in a country that is rapidly growing more diverse in a globalizing world. We consistently see this phenomenon in surveys showing that large numbers of white people believe racial discrimination against them is as pervasive, or more so, than it is against African Americans.

It’s the same dynamic that researchers at Harvard Business School described in a recent study: White people tend to see racism as a zero-sum game, meaning that gains for African Americans come at their expense. Black people see it differently. From their point of view, the rights pie can get bigger for everyone.

Black Lives Matter is not a hate group. But the perception that it is racist illustrates the problem. Our society as a whole still does not accept that racial injustice remains pervasive. And, unfortunately, the fact that white people tend to see race as a zero-sum game may actually impede progress.

Originally published by Time