You can make 2 querys to get both groups of users that you want. I mean:
1) Users who dont have any role.
User::whereDoesntHave('roles')->get();
2) Users have any role, but not admin (from Adam Kozlowski answer)
$this->users()->whereHas('roles', function ($query) {
$query->where('name','!=', 'admin');
});