An Analysis of Cuyahoga County

Judicial Candidate Rating System and Process

By
Gregory L. Brown, MSUS
And
Elaine Sutton, Ph.D.
Brown and Associates Consulting Services
 Brown and Associates Consulting Services

• Founded in 2008
• Gregory L. Brown, MSUS, President
• Elaine Sutton, Ph.D., Senior Research Associate
• Expertise :
– Strategic thinking and planning
– Project and program development and management
– Organizational development and leadership
– Community building and organizing
– Meeting Facilitation
– Public policy and advocacy
– Governmental relations
– Survey research, data analysis and evaluation

2
 Brown and Associates Consulting Services

• Client List
– The Cleveland Foundation
– Cleveland State University Levin College of Urban Affairs
– Prevention Research Center for Healthy Neighborhoods, CWRU Medical
School
– Burten, Bell, Carr Development, Inc.
– Environmental Health Watch
– Faith Community Credit Union
– Cuyahoga County Board of Health
– African American Collaborative Obesity Network (AACORN), Drexel University

3
 Judge4Yourself.com Overview

• Brown and Associates Consulting Services conducted an independent

review of the Judge4Yourself.com Cuyahoga County judicial candidate
rating system and process.
• Judge4Yourself.com is a services of the Judicial Candidate Rating Coalition
formed to provide information and perspectives for voters.
• Judge4Yourself.com attempts to convey in simple language whether or not
they believe a candidate will make a good judge.
• The organizations involved in Judge4Yourself.com are:
– The Cleveland Municipal Bar Association
– The Norman s. Minor Bas Association
– The Ohio Women’s Bar Association
– The Cuyahoga County Bar Association

4
 Judge4Yourself.com Overview

• Judge4Yourself.com uses a judicial candidate rating process to evaluate five

criteria:
– Judicial Temperament refers to the candidate's ability to conduct him/herself
in a manner that is dignified, tactful, respectful, objective, patient, open-
minded and compassionate. This criterion also refers to a candidate's ability to
work well with other judges and court officials; be willing to listen to
unpopular ideas; and work to reframe from coercing settlements
– Integrity refers to the candidate's ethics, character, and his/her reputation in
the community. The group is specifically looking for a candidate’s ability to
show impartiality and independence; trustworthiness; compliance with the
law; and a general commitment to equal justice under the law.
– Diligence refers to the candidate’s ability to be reliability; punctuality; pre-trial
and hearing preparedness; and efficiency. The group is also trying to ascertain
if candidates can issues decisions in a timely fashion; use resources and court
time wisely and comply with the Rules of Superintendence.

5
 Judge4Yourself.com Overview

– Community Involvement and Understanding refers to working as a teacher at

a law school; being an active member of the bar association; doing pro bono
work and community or public service; and diversity of life experiences.
– Professional Competence refers to legal knowledge; sound professional
judgement; courtroom experience and absence of prejudice based on gender;
race; age; disability; religion; sexual orientation; marital status; or political
affiliation.

6
Judge4Yourself.com Data Collected and Analyzed

• May 8, 2018 Primary Judicial Ratings – Cuyahoga County Court of Common Pleas –
General Division
• Rating for 2017 Judicial Ratings – Municipal Courts for Cleveland, South Euclid,
Lyndhurst and Cleveland Heights;
• Ratings for Ohio Justice of the Supreme Court 2016
• Ratings for Judicial Candidates 2016, Local Elections - Court of Common Pleas – General
Division – Slot 1
• Ratings for Judicial Candidates 2016, Local Elections - Court of Common Pleas – General
Division – Slot 2
• Ratings for Judicial Candidates 2016, Local Elections - Court of Common Pleas – General
Division – Slot 3
• Ratings for Judicial Candidates 2016, Local Elections - Court of Common Pleas –
Domestic Relations – Slot 1
• Ratings for Judicial Candidates 2016, Local Elections - Court of Common Pleas –
Domestic Relations – Slot 2
• Ratings for Judicial Candidates 2016, Local Elections - Court of Common Pleas –
Domestic Relations – Slot 3

7
 Judge4 Yourself Data Analysis Results

• Of the nine sources of data collected for analysis, only four had data to compare
African and Caucasian American candidate rating scores.
• Of the four data sources used for comparison, no matter the judicial court position
being sought, African American candidates received a lower overall mean rating
score than their Caucasian counterparts.
– Court of Common Pleas Primary Election, 2018 – Caucasian Candidates, 2.6,
versus African American Candidates, 1.63
– Municipal Courts (Cleveland, South Euclid, Lyndhurst, Cleveland Heights, 2017
– Caucasian Candidates, 3.1, versus African American Candidates, 1.7
– Court of Common Pleas General Division-Slot 3, 2016 – Caucasian Candidates,
4.0, versus African American Candidates, 2.25
– Court of Common Pleas Domestic Relations-Slot 3, 2016 – Caucasian
Candidates, 3.75, versus African American Candidates, 3.0

8
 Data Collection and Analysis Findings

• Finding - Based on the limited amount of data used for this analysis, the consistently
lower mean rating scores received by African American judicial candidates is a cause for
concern.

9
 Recommendation

• As researchers, we reasonably ask what causes the differences in the ratings

between African and Caucasian American candidates.
• For the purposes of our analysis we only had access to a limited amount of data.
So it is difficult to make any general statements about the data collected and the
corresponding analysis.
• But our analysis of the data collected, based on a point in time is troubling in an
environment where the community climate is trending towards greater diversity
and inclusion in all aspects of our society’s public and civic system and structures.
• We believe further study and research is needed to discover if this point in time
analysis is representative of a deeper and more significant trend in the rating of
judicial candidates.

10
 Recommendation

• There are a number of questions that need to be considered and addressed by impartial
research and investigation to ascertain if the ratings process is fair and just for all
candidates.
• Some of the questions to consider are:
– Are all African American candidates seeking judicial positions in Cuyahoga County
unprepared and unqualified to serve? If so, why are they determined to be unprepared
and unqualified?
– Is the rating system and process being used biased?
– Is the rating system and process objective?
– Are the criteria and process too subjective, causing each person in the rating process too
much latitude in assigning rating scores and deciding who will be recommended and
who will not?
– Are there criteria or other factors, other than those being used as criteria that might be
better suited to rate candidates and would provide a more equitable basis for decision-
making?

11
 Analysis of Candidate Questionnaire

• After conducting the analysis of the Judge4Yourself.com judicial candidate rating

data we were asked to review and analyze the Judicial Candidate Questionnaire
and rating process.
• The Judicial Candidate Questionnaire is used as the common rating system
Judge4Yourself.com.
• Based on the data collected from questionnaires, candidate interviews and bar
members’ discussions, candidates receive a rating score and ultimately a final
candidate determination.
• There are five different final determinations a candidate can receive.
• They are: excellent, good, adequate, not recommended or refused to participate.
• The final determination of refused to participate does not apply if the candidate
participates in the Judge4Yourself.com interview.

12
 Judicial Candidate Questionnaire
• The Judicial Candidate Questionnaire is used to capture primarily qualitative and some quantitative
data about prospective judicial candidates. The questionnaire seeks general information about the
following:
• Candidates’ educational background;
• Military service;
• Work history and professional admissions;
• Honors and awards;
• Bar association and other memberships;
• Written work products;
• Judicial experience;
• Public office;
• Law practice;
• Litigation experience;
• Other law related activities;
• Potential conflicts of interests;
• Contacts;
• Past conduct;
• Health
• Goals and qualifications.

13
 Analysis of the Candidate Questionnaire

• Based on our review of the Judicial Candidate Questionnaire and rating process; discussions
with current African American Cleveland Municipal Court Judges; and bar member
representatives who have participated in the candidate rating process, we gained insights
about the use of the questionnaire and corresponding rating process. These insights
included:
• Candidate questionnaire responses are used to determine the questions candidates will be
asked.
• Any bar association member representing a bar membership group comprising
Judge4Yourself.com can ask questions of any candidate.
• The questions asked each candidate are not the same or uniform.
• Each bar association group comprising Judge4Yourself.com convenes to discuss each
candidate’s questionnaire and interview responses.
• Each bar association, based on their discussions develops a rating score for each candidate.
• All bar associations reconvene to discuss their scores.
• The scores of each bar association are totaled and an average rating score determined.
• Based on the average rating score, candidates are determined to be excellent, good,
adequate, not recommended.
• Only those candidates who are not interviewed receive a rating of refused to participate.

14
 Findings

• Finding #1 - We concluded that the rating system and process used by

Judge4Yourself.com to assess judicial candidates is very subjective.
– Cleveland Municipal Court African American judges and former
Judge4Yourself.com participants raised concerns about the objectivity of the
rating system and process.
– The final determinations for candidate ratings are based primarily on
qualitative data gleaned from candidate questionnaires and interviews.
– The design of the candidate questionnaire is open-ended.
– Meaning that no options are provided to respondents to answer questions.
– The information received from the open-ended candidate questionnaire
results in qualitative data.

15
 Findings

– Qualitative data is information gathered that is descripted and not based on

numbers.
– These types of data are not measurable, but rather illustrative and categorical
in nature.
– Because qualitative data is subjective in nature, it is less reliable compared to
quantitative data.
– When qualitative data is pooled together, coded and analyzed, it could
become quantitative data.

16
 Findings

• Findings #2 - The candidate review and assessment process is much more subjective
than objective, which directly influences rating scores and final determinations.
– Each bar association has the latitude to craft questions for candidates based on
their assessment of the candidate’s questionnaire responses to specific questions.
– So the questions asked of candidates are not uniform in the interview portion of
the rating process.
– The questions asked candidates not being uniform allows those deciding rating
scores an opportunity to be influenced by their subjectivity and the subjectivity of
other bar members leading to biased results.
– We were unable to discern if there is a corresponding rating instrument that allows
raters to quantify numerically the data collected in a uniform, non-subjective and
unbiased manner.
– This rating instrument would pool the qualitative data collected together and code
the data for statistical analysis.
– This conversion would make the rating process more objective.

17
 Findings

• Finding #3 - We were unable to understand the connection between the review

and discussion of the qualitative candidate data collected and how it is translated
into a quantitative rating scores for candidates.
• Finding #4 - We are also unable to understand the connection between the
candidate rating scores and how the final determination of excellent, good,
adequate, not recommended, refused to participate are made.
– For the purposes of this analysis, objective means dealing with facts as
perceived without distortion by personal feelings.
– The facts should ultimately result in quantitative data.
• Finding #5 - We believe that overall the candidate rating process needs to be re-
evaluated to assure that the process being used is as objective as possible.

18
 Findings

• Finding #6 - Based on the data analysis conducted of rating score given to African
and Caucasian American judicial candidates; the candidate questionnaire and
rating process, we see the combination of subjective decision-making and implicit
cultural bias in the candidate rating system and process.
– Implicit bias occurs when someone consciously rejects stereotypes, but also
holds negative associations in his/her mind unconsciously.
– Implicit bias does not mean that people are hiding their racial prejudices.
– People literally do not know that they have them.
– Yet, researchers have concluded that the majority of Americans hold some
degree of implicit bias.
– Cultural bias is interpreting and making judgements based on standards
inherent to one’s own culture.
– The data collected and analyzed, though it was a limited sample, indicated
that African American candidates consistently receive lower rating scores than
their Caucasian counterparts.

19
 Findings

• Finding #7 – The criteria used by Judge4Youeself to evaluate the qualifications and

character of judicial candidates is subjective.
– The criteria definitions are open to broad interpretations by bar members.
– Being open to broad interpretations, the criteria is not applied to each
candidate the same way.
– Broad interpretations in how the criteria is applied leads to broad variations in
the candidate rating scores.
– There is no explanation of the weight given each criteria in the rating process.
– Again this allows significant latitude and subjectivity in decision-making
among bar members and bar associations.

20
 Recommendations

• Recommendation #1 – Conduct an independent review to assess the complete

judicial candidate review and rating system and process.
• Recommendation #2 – Require complete transparency in a newly created judicial
candidate rating system and process that is objective and unbiased.
• Recommendation #3 – Require the newly created judicial candidate rating system
and process to clearly explain how the activities and tools used translate
qualitative data into quantitative results.
• Recommendation #4 – Create a new judicial candidate rating system and process
that is more inclusive of other community stakeholder voices that are not currently
participating in the process. Creating a more inclusive process may prevent a rating
process that is both subjective and culturally biased.

21
 Recommendations

• To make the data collection elements of the rating process more objective, the
creation of an instrument that translates qualitative data into quantitative data is
recommended.
• We note that one explanation for the consistently lower rating scores for African
American candidates is that they are unqualified and unprepared to hold the
positions they seek.
• Because of this possible explanation, we strongly recommend that additional
research by an independent researcher is conducted to vet this question fully for
the benefit of all involved.
• We suggest that the research is an immediate remedy to address if the cultural
bias is a result of implicit bias inherent to those engaged in the rating process

22
 Thank You

Question and Comments

23