According to The Washington Post, Carson made the comment during a HUD meeting in San Francisco, saying “big, hairy men” are infiltrating women’s homeless shelters.
Carson also said society no longer seemed to know the difference between men and women, comments which upset many of the staffers at Tuesday's meeting and caused one woman to walk out, The Washington Post reported.
“The Secretary does not use derogatory language to refer to transgendered individuals. Any reporting to the contrary is false," a HUD senior official said in a statement to The Washington Post.
The official added that Carson was referring to men who pretend to be women to gain access to battered women’s shelters and not singling out transgender women as “big, hairy men.”
It's not the first time Carson has gotten himself into trouble with his comments on transgender people.
Carson has previously said transgender people are abnormal and they shouldn't be in the military.
In an interview with The Hill in 2016, Carson said being transgender “doesn’t make any sense” and compared it to changing ethnicity.
“For someone to wake up and think that they belong to a different sex because they feel different that day is the same as if you woke up and said, 'I’m Afghani today because I saw a movie about that last night, and even though my genetics might not indicate that, that’s the way I feel, and if you say that I’m not, then you’re a racist,'" Carson told The Hill.
Carson also compared being transgender to a leopard trying to change its spots in his interview with The Hill.
“It’s not based on your feelings, it’s based on what you are,” Carson said. “Can a leopard change its spots? No. It can’t because it feels like maybe it’s a different creature. No, it’s still a leopard. Sorry.”