rmdir(2) should return ENOTEMPTY

Under Solaris the behavior for rmdir(2) is to return EEXIST when
a directory still contains entries. However, on Linux ENOTEMPTY
is the expected return value with EEXIST being technically allowed.
According to rmdir(2):

ENOTEMPTY

pathname contains entries other than . and .. ; or, pathname has
..  as its final component.  POSIX.1-2001 also allows EEXIST for
this condition.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #895