Most creatures do not defecate where they eat, an instinct that keeps fecal-borne illness to a minimum in most populations. Humans, in general, do not use their own waste as fertilizer because of the ick factor. It's o-kay to use cow waste, but if it came from your own self, it's disgusting. It is reasonable to assume that if the giants have developed culture, then they may have similar taboos.

Then again, they may not.

In any case, there is no evidence of giants farming recorded in anything I've read. I'd have to do more research to see if there are any references out there that contradict what I know so far.