A frag is a point awarded a player whenever they kill another player. On some servers, a match will end after a player acquires a certain amount of frags, decided upon by the server using the cvar fraglimit.
Frag is short for fragmentation, as in fragmentation grenade. Because of the general insanity of the Vietnam war, a lot of infighting happened. The "accidental" killing of a friendly superior officer with a fragmentation grenade during the confusion of a jungle firefight was referred to as "fragging". For example, "I didn't mean to frag LT, I was throwing for the goddamn Charlies, I swear!"
Comically enough, id Software chose to use the term for deathmatch kills in Doom, presumably because you are killing fellow marines. The term came to be associated with gaming in general, whether you're killing marines or not, with grenades or anything else.