District 15 precinct committee members Tuesday voted unanimously to urge Patterson to quit at the conclusion of a three-hour meeting, most of which was behind closed doors at a Boise retirement home.
Two policemen from Boise were posted outside after some members of the committee expressed concerns for their safety.
There were no incidents, and Patterson didn't attend the meeting, despite an invitation.
In a telephone interview Tuesday before the vote, Patterson said he "knew it was going to be a circus" so he opted not to accept the panel's invitation for him to come and explain his actions.
He didn't respond to a request for comment after the no-confidence vote.
The committee's vote is only a request; it can't remove him.