HOM stands for Hall of Mirrors. It's a effect which often occurs when there is no texture. It appears in the game as freak effects.
HOM Description in Unofficial Wad Designer's Guide