Being an individualist does not require me to undertake the impossible task of trying to assess the individual character of every single person I might ever encounter in life. It's not wrong to take into account the relative probabilities entailed by some aspect of a person's identity. It's called critical thinking. To fail to engage in it over concerns of how others might perceive my conclusions would be to bury my individuality.
If you believe that race plays no role in intelligence or athleticism or anything beyond "skin pigmentation," you have either never ventured out into the world or are massively evading the realities it presents. I suggest that you read Charles Murray before concluding that libertarianism cannot accommodate a realistic perspective on racial differences.
The U.S. prison population is more than 40% black, even though blacks are only 13% of the total U.S. population. Whites are less than a third of all prisoners even though they are 66% of the population.
I accept that other factors besides race may help to explain part of the racial disparity in crime statistics. But if crime data across cultures, continents, and economies continue to all point to a correlation between race and crime for the next 1,000 years, would that be enough to open your mind to consider with seriousness what Darwin and Jefferson hypothesized generations ago based on their much more limited observations -- that race has something to do with behavior?