Geek points are awarded or removed by your peers. Usually point levels are only changed when the geek has done something which is significant in the context of their current geek standing and their perceived geek potential.
Successfully configuring an email client is worthy of an award of geek points when completed by someone who has few geek points, but the same action by an accomplished geek is just unremarkable noise. It is possible to gain points by making a particularly clever joke, but it's a razor's edge - there's a very thin line between super witty and appallingly bad, which can lead to point removal.
Losing geek points is a shameful event that should be avoided if possible. There are many ways to manage it, including but not limited to failing to catch an obvious geeky reference or doing something spectacularly stupid in normal geek duties.
In extreme cases, a geek card may be revoked, which Marv has done to Geoff on more than one occasion. This once happened because he admitted that he did not know what a Con was.