Measuring the effectiveness of mentoring

David Clutterbuck’s seminal study of the dynamics of mentoring relationships is based on an extensive longitudinal study of mentoring pairs, with data gathered from both mentors and mentees. The study, which aimed to explore the effectiveness of developmental mentoring, had four main parts:

• Literature search and focus group activity to identify potentially critical variables on the efficacy of structured mentoring
• Testing of items to establish metrics and validate scales
• Re-testing the scales and questionnaires on a sample of matched pairs from none organisations
• Analysis of data and drawing of conclusions regarding cause and effect phenomena.

The pilot study involved non-matched pairs at three points in time. Time 1 was at the beginning of relationships, with questions relating to goal orientation and participants’ expectation of their own and each other’s behaviours during the relationship. Time 2 was after 6 months, with questions relating to participants’ perceptions of their actual behaviours and those of their dyadic partner; and to relationship satisfaction. Time 3 again explored relationship satisfaction and asked for information about outcomes for both parties. The numbers in the pilot study are shown in the box below.

Factor analysis produced a number of scales with acceptable to high Cronbach’s alpha. Goal orientation became three measures: goal clarity (what they wanted to achieve), goal alignment (a sense of shared purpose), and goal commitment (how committed they were to achieving their individual objectives). Three behavioural measures emerged for developmental mentoring for mentors and three for mentees – self-development, self-reliance and openness to counselling, on the part of the mentees; and, on the part of the mentors, support for the mentees’ self-development and self-reliance, and provision of counselling. These measures contrasted strongly with negative scales representing behaviours associated with sponsorship mentoring – the first practical demonstration that these are indeed two separate constructs.

Three measures relating to the “feel” of the relationship emerged: relationship satisfaction, relationship quality and relationship commitment. Finally, the four proposed categories of outcomes demonstrated strong scales, with minor modification.

The main study used matched pairs, which allowed for comparisons within the dyads and followed the same pattern of sample times, using a database of 62 pairs responding at both T1 and T2; and 28 at T1, T2 and T3.

1.  Goals

Goals in the mentoring relationship are important in focusing the learning conversations. There are three key elements to these goals:

• goal clarity (what I want to achieve and why?)
• goal commitment (how determined I am to achieve the goal?; how important they are to me?)
• goal alignment (to what extent are the goals shared and mutually understood? Is there a sense of purpose to the relationship?)

Having strong goal clarity and commitment at the beginning does not necessarily lead to either a better relationship experience or positive outcomes for the mentee. However, having a sense of purpose at the beginning is associated with both relationship experience and mentee outcomes. It seems that specific goals, which have a positive impact on behaviours, experience and outcomes need time to evolve; and that initial goals are often adapted considerably as a result of the mentoring conversation. Parallel research around career pathing (Ibarra H & Lineback K 2005 ‘What’s your story?’, Harvard Business Review, 83(1).) suggests that too close a focus on specific objectives is counterproductive. What counts is being attentive to opportunities in the working environment and a general sense of direction.

The goal scales are made up of the following questions:

• Clarity (Mentor alpha: 0.84; mentee alpha: 0.88)
  • I know exactly what I want to achieve in my career development
  • I know exactly what I want to achieve in my personal development
  • I know how to achieve my career development goals
  • I know how to achieve my personal development goals
• Commitment (Mentor alpha: 0.57; mentee alpha: 0.65)
  • I am committed to achieving my career goals
  • I am committed to achieving my personal development goals
• Alignment (Mentor alpha: 0.91; mentee alpha: 0.80)
  • I know what I want out of the relationship
  • I know my mentor’s (mentee’s) goals for the relationship
  • I know the organisation’s goals for the mentoring programme

So why measure goals at the beginning?

The primary value of measuring gaols at the beginning is to test for alignment/ sense of purpose. The average of the mean scores for the alignment scale is xxx. Measuring goal clarity and commitment provides a useful indicator of how much thought the participants have given to the relationship in advance. A low score might suggest that participants weren’t sufficiently prepared; or that they are content to allow the specific goals to emerge over time. Interviews with a small sample of participants will normally identify which of these reasons applies. If you wish to explore the issue of goals more thoroughly, you can repeat the goal scales after six months or at the relationship mid-point.

2. Behaviours

Our analysis of mentoring behaviours reveals clear distinctions between developmental and sponsorship mentoring. In the original research, participants were offered a range of statements, some reflecting the sponsorship mentoring literature, some the developmental mentoring literature. There was very little cross-over between the two constructs, indicating that they are indeed different kinds of relationship.

Sponsorship mentoring places greater emphasis on directive behaviours by the mentor (doing things on the mentee’s behalf, making use of the mentor’s authority), while developmental mentoring emphasises non-directive behaviours (helping the mentee do things for him/herself and using questions to help create and use insight). In practice, all mentoring relationships operate along a continuum of directiveness. At times, the mentor may need to advise and initiate discussion topics; at others, the mentee takes charge of the process.

Mentoring behaviours can also be viewed along the continuum from stretching (challenging) to nurturing (supporting). Again, the mentor needs to choose according to circumstances, where to pitch his or her intervention. The behaviours covered in the MDS all fall within a matrix of these two continua.

The scales that emerged from the pilot studies are:

Directive / non-directive

Self-development (mentee)/ supporting self-development (mentor). Mentor and mentee alphas exceeded 0.80, measured through both self-perception and perception of each other, both in terms of expectation of behaviour and actual behaviours at the mid-point.

As a mentor, I expect to:

• Ask good questions
• Encourage mentee to put self forward
• Encourage mentee to do things for self
• Encourage mentee to set own stretching goals
• Help mentee develop networks
• Give benefit of my experience when needed

(Mentee questions parallel these.)

Stretching/ nurturing

Self reliance (mentee) / supporting self-reliance (mentor). Again, alpha scores exceed 0.80

As a mentee, I expect to:

• Be open to suggestions re raising my ambitions
• Face up to difficult issues
• Explore assumptions about myself
• Seize opportunities to stretch myself
• Step outside my comfort zones

(Parallel questions for mentors.)

Counselling (mentor)/ openness to counselling (mentee). Again, alpha scores exceed 0.80

As a mentee, I expect to:

• Bring to discussion personal and work issues
• Seek understanding of my motivations
• Use the relationship to develop a network of support

(Parallel questions for mentors.)

MDS examines expected behaviours at the start of the relationship (providing a check that people understand what is expected of them) and actual behaviours at the mid-point (providing a check that both parties are behaving in ways appropriate to strong and effective mentoring). The data from MDS can be used to:

• Stimulate discussion between mentor and mentee about their expectations of each other and their experience of how each is contributing to the learning conversation
• Provide input into review sessions, where participants meet to discuss problems and acquire additional techniques / skills

3. Relationship experience

A number of statements relating to relationship experience were gathered from the mentoring literature and from participants in mentoring programmes. These gave rise to three scales:

• Relationship quality. (Mentor alpha: 0.81; mentee alpha: 0.90)

How satisfied are you with:

• The meeting environment
• Your freedom to influence the agenda
• Your opportunity to explore issues
• How the relationship is managed
• The value placed upon your opinions
• The time you are both able to commit to the relationship
• The variety and scope of the discussions
• Your confidence the relationship will last

• Relationship satisfaction. (Mentor alpha: 0.81; mentee alpha: 0.84)
  • We have come to know and understand each other
  • We share the same values
  • We are relaxed that discussions remain confidential
  • My mentor/ mentee is well disposed towards me
  • We get on well together
• Relationship commitment. (Mentor alpha: 0.72; mentee alpha: 0.93)
  • I am strongly committed to this relationship
  • I am prepared to invest substantially in this relationship
  • I see the relationship as a great opportunity
  • I am pleased to have been chosen
  • This is the right mentor/ mentee for me at this time
  • I feel loyalty to my mentor/ mentee

The item relating to being a great opportunity tends to score higher for mentees than mentors.

Together these scales provide an overview of the relationship. Programme coordinators can use these measures to compare:

• Mentor v mentee perceptions of how the relationship is going
• The relationship experience at mid-point and after 12 months.

4. Outcomes

Four categories of outcome were revealed from the data analysis:

• Developmental (which may include changes in knowledge, technical skills and behavioural competence) (Mentor alpha: 0.82; mentee alpha: 0.88)
• Career – achievement of career goals (Mentor alpha: 0.92; mentee alpha: 0.90)
• Enabling (for example, having a career plan, a self-development plan, or more extensive support networks). (Mentor alpha: 0.83; mentee alpha: 0.81)
• Emotional (changes in emotional state, such as increased confidence, altruistic satisfaction, enjoyment of the intellectual challenge). (Mentor alpha: 0.83; mentee alpha: 0.85)

It is typical that career outcomes will score lower than other outcomes. This is in part because 12 months may not be sufficient time for changes to occur (especially if the stimulus for mentoring is a recent change of role); and in part because developmental mentoring places greater emphasis on personal growth as a stepping stone towards career progress, while sponsorship mentoring emphasises career progression more directly. Mentor perception of their outcomes score slightly lower than do those of mentees. A small gap here should be of no concern. However, a large difference between mentor and mentee scores (more than 1.5 out of 5) may indicate that mentors are not using the potential of the relationship for their own learning.

5. Norms

All of the questions are reproduced below, along with:

• Average scores of all mentors from all organisations
• Average scores of all mentees from all organisations
• Median scores of all mentors from all organisations
• Median scores of all mentees from all organisations

© David Clutterbuck, 2016

Prof David Clutterbuck
Coaching and Mentoring International Ltd
Woodlands, Tollgate,
Maidenhead,
Berks, UK. SL6 4LJ

www.coachingandmentoringinternational.org
e-mail: info@coachingandmentoringinternational.org
Company registration number : 08158710