[GFS2] Don't try to remove buffers that don't exist