Whitney Houston -- Family Party Turns Into Raging Warfare

Whitney Houston's son-in-law got his ass beaten by one of Whitney's nephews at a formal family gathering ... and it's all because of a long-simmering family feud ... TMZ has learned.

We're told Whitney's nephew, Gary Michael Houston, went nuclear on Bobbi Kristina's husband, Nick Gordon … allegedly SLAMMING him to the floor and pummeling his face -- accusing him of talking crap about the family.

Family sources tell us ... it all went down at a Sweet 16 party for Whitney's niece, Rayah. We're told it was a formal affair -- everyone in tuxes and long gowns -- but Nick showed up in a hoodie and knit cap and "lookin' like a thug."

Gary felt Gordon had insulted the family one too many times ... as soon as the party was over, he delivered the beat down. We're told it was a lopsided fight -- Nick didn't even get in a single punch.

The two are now sluggin' it out on Twitter. Gordon posted, "F*** all of em they've tried to fight me (Gary H) jealous ass n***a."