This staff guidance page will help LJMU staff use BuddyCheck software to support group learning. Please note this software is on a 12 month evaluation 2020-21 

What is it?

BuddyCheck is an easy to use peer evaluation tool within Canvas, which allows students to evaluate each team member’s performance in a group activity, as well as self-assessment. It is a valuable tool for helping your students successfully manage group work.

Advantages  

LJMU students undertake formative and summative group work depending on the design of their programme. Group work has been shown to promote deeper, more active, cooperative, and experiential learning that helps develop social skills and learning community networks, which are both being useful within the world of work.  However, the most common complaint from students is parity between student contribution, and students can also struggle with learning and implementing team management skills [1,2,3]. We recommend you use this for both formative and summative group work tasks. Demonstrate the software to the students to help them understand how this will help them improve their group working skills and outcomes. Tell the student how you will be using the software to identify groups with issues so that you can help overcome them.  

1. It will provide you with a useful window into what is happening within your groups. 
2. Your students will be more motivated to participate and act professionally because they will know the process is fairer. This will reduce the number of complaints. 
3. Assessment will be fairer through the use of the student feedback as the final grade of the individual student can be adjusted using the results obtained in the evaluation.
4. It will save you time 
5. It is a good aid when providing students with feedback on their assessment. 
6. Main Functionalities 
   1. You can create rubrics yourself or use the templates provided  
   2. Extensive results table, including labels with common scenarios per student such as 'Underconfident' or 'Disagreements' 
   3. Every student receives an adjustment factor which you can use to adjust the final grade 
   4. Anonymous results for students with tips on how to improve 
   5. You can remind students to complete the evaluation in one click 
   6. Students can send private feedback messages to you
   7. Adjust final grades and import them into the Canvas grades area.

Buddycheck supports your marking, not replaces it

It is important for you and your student to understand that the students are not marking each other. Their responses are useful information to use to guide any final mark you make for each individual student. At each stage of the process, you are in control of the marking process and can adjust the marks before they are released to the students.

How does it work? 

The BuddyCheck process is in the following three parts.  

1. Creating a BuddyCheck evaluation 

Watch this video guide to help you understand the process of creating an evaluation. Here is a step by step guide.   

1. Go to Canvas 
2. Click on a module 
3. Add group configutations to your Canvas course.  
4. Click on the BuddyCheck menu item in the Canvas course menu. If BuddyCheck is not visible in the Canvas course menu in your Canvas course, go to settings and the navigation tab and add it to the active menu list and save.  
5. Click 'Create evaluation'. You will perhaps see a verification screen asking you to confirm connecting your account temporarily to BuddyCheck. This way, BuddyCheck can pull the group and student data from your course on behalf of you. 
6. This will load the BuddyCheck page. From here you can: Create, modify, monitor or delete evaluations  
7. When creating an evaluation you can:  
   • Choose a name for the evaluation  
   • Choose the Start and end date of the evaluation  
   • Choose (on / off) which questions should be used in the peer evaluation.  
   • Select (on / off) team questions: Additional questions about the functioning of the group as a whole.  
   • Switching (on / off) private comments towards the instructor  
   • Preview the evaluation as a student  
   • Feedback: The instructor can ask students to write personal feedback towards each group member. This feedback is only visible to the receiving student and not anonymous.  
   • Click 'manage questions' in the middle of the form and a dialogue opens where you can create new questions, edit those questions and even import custom questions from an old course you were involved in. Our advice: Use default questions. The default evaluation method is designed by the people behind CATME .They have years of experience with this evaluation method and are convinced that using this method consistently, offers the best results. In addition, allowing instructors to create their own questions requires more time. Because creating good questions requires time and testing. In order to offer an easy to use peer assessment solution, we thought it would be best to keep it simple and consistent with a proven method. 
   • Once you have created your first evaluation, click the Results button of this evaluation to see if all groups are added correctly. 
8. When an assessment is created, students receive an automated (customisable) email to complete the survey.  

2. During the evaluation 

Watch this video guide to help you understand the process during an evaluation.  

When the start date has passed, all students will receive an email from BuddyCheck requesting them to evaluate their group. Each email has a unique link that enables the student to get started without having to go to Canvas first. However, it is also possible for students to open their evaluation from within the Canvas course and clicking the BuddyCheck tab. As an instructor, you will see the progress and on the Results page of the evaluation, you can monitor whether your students are filling in the evaluation correctly. If they do not complete in the time you can ask students to redo their evaluation. 

The student fills in a number of topics for himself and his peers. 

 

To monitor students during the evaluation 

1. Go to Canvas 
2. Click on a module 
3. Click on the BuddyCheck menu item in the Canvas course menu.  
4. The BuddyCheck interface will open, from here you can: 
5. See the status (Pending, Started, Closed, Locked, Released) at the start screen  
6. See the percentage of students who have completed the evaluation  
7. Click the Results button to view a table of all results. The table contains the following information:  
   • Student’s name  
   • Groups’ name  
   • Note labels: High performer, Low performer, Underconfident, Overconfident, Manipulator, Cliques, Conflict. These provide direct feedback about the performance of the student and/or the group. 
   • The average score of the student 
   • Adjustment factor (score that can be used to correct a final grade of the student) with and without self-score.  
   • Date of submission  

Actions you might need to take during the evaluation 

In the results tab, you can select one or multiple students in the results table using the checkboxes on the left. By doing so, action buttons will appear on the top of the results page: 

Send an email to students who haven’t completed 

Use this button to email students. The tutor can send a reminder to all students that have not yet submitted the evaluation. Click on the evaluation title and then you see a button 'More'. Click the button and click 'Send a reminder'. This will send an email with a link to the survey to each student who hasn't completed the survey yet. 

Undo or redo an evaluation 

Use the undo or redo button if you want students to fill out the survey again 

Remove a student 

Use the remove button if students have left the course in the meantime. (Remember to also remove them from the groups in the Canvas course itself, otherwise, they will be added again when you create a new evaluation). 

How to change groups or students in existing evaluations 

Simply change the groups in Canvas groups area. You can synchronize groups and students after the start of the evaluation without losing any results. In case of a change, students are only asked to update their evaluation if the change applies to them.  

Go to the evaluation you created and click 'More' > 'Sync with course'. BuddyCheck will then ask for permission and get all students and groups and make changes accordingly. Only in scenarios where a student has to do the evaluation again, an email will be sent.  

View of the results 

You can view all individual results in a table. Clicking on a student name shows more detailed data. Labels such as 'Underconfident' or 'High performer' quickly provide useful information for you. The final adjustment factor can be used to adjust the grade of a student in the Canvas grades area.  

Can students submit after deadline? 

Yes, students can still submit after the deadline. Their submissions will be marked red to indicate they were late.  

If you want to stop submissions after the deadline 

There are two options to make sure students cannot submit anymore: Lock evaluation or release results. 

Lock evaluation 

Go to the results page and click 'More '> 'Lock evaluation'. Students cannot submit anymore, even if you reminded them or reopened the evaluation for some students. 

Release Results 

Go to the results page and click 'More '> 'Release results'. Once the results are released to the students, it not only becomes locked but you also send an email to all students with their results. You can see a preview of each student result page. Go to the results page and expand a student row in the table and click 'Preview result page'. 

Can I reopen the evaluation of individual students? 

Yes, you can! Assuming you haven't locked or released the results yet. If you locked the results, you have to unlock the evaluation first. 

You go to your evaluation results where you see all students in a table. On the left of each student is a checkbox. Select each student evaluation you want to reopen and on the top of the page buttons appear, one of them being 'Reopen'. Click on it and optionally tune the email text and send it.  

Students will directly receive the email. With the link in it, they can double-check or adjust their evaluation and submit it again.  

If you already released the results but really need to reopen the evaluation for some or all students, please contact support using the chat inside the BuddyCheck interface. 

Can students give direct feedback to their peers? 

Yes, this is possible. The functionality is called 'Peer messages'. Peer messages allow you to ask your students to send optional feedback to each peer directly. Only the student that received the feedback will see the message if you release the results and it will not be anonymous. You can also see these messages. 

3. Import to Canvas grades  

If you want to use the marks for a summative assignment please use the following method.  

In Canvas 

1. Set up a group assignment in Canvas. 
2. Set the assignment up so that you are marking the students as a group 
3. Mark the students as a group 
4. Change the setting on the assignment to mark the students individually 

 

In Buddycheck 

1. Go to the BuddyCheck assignment you created and the students have completed 
2. Remove any students who are no longer taking the assessment (see below)
3. Click the More button 
4. Select Publish grades 
5. Select the assignment  
6. Select Continue 
7. Adjust the weighting factor and review the effects of the grades 
8. reduce the decimals if needed
9. Select Accept and Continue 
10. Select Download backup which will keep a record of your previous marks and help you if you want to restore the grades 
11. Select Update existing grades 
12. In the Canvas grades area, note that the students will receive a comment telling the student that their grade was adjusted and by how much. 
13. Consider changing any grades and feedback relating to the change in grades made by Buddycheck

Remove students

The student groups may contain students who have either left the course during the group work, or who have not taken part in the group activity because of illness, or other reasons. Leaving these in the BuddyCheck process could distort the adjustment factor calculations in such groups. These students can be identified by the fact that they received an average score of 1-2 and probably have not submitted themselves. We recommend that you remove these students from BuddyCheck. This can be done by clicking on the checkbox next to their names and using the remove button at the top of the column. Please note once removed they cannot be readded to this BuddyCheck evaluation. It is recommended to remove them from the group set in Canvas as well.

After the evaluation 

When the evaluation deadline has passed, the evaluation is formally closed. However, students can still submit their survey. They will be marked red to indicate they were late. As soon as you are happy with the results, you can release the results to the students. After release, all students will receive an email with a link to look at their individual result. Students who haven't submitted the survey will also get the results. Students will not see the individual rating from group members. Just a total average of all group members. 

About the adjustment factor 

A ‘weighting factor’ is generated for each individual group member which is derived from each student’s input against defined criteria. Based on the total mark given to the group task, assessed and allocated by the academic tutor in the usual way, the weighting factor is then used to moderate marks, providing an individual mark for each student. 

This value is meant to be used as a way to evaluate how well students can work in teams. This factor can be used to adjust the grade of a student in the Canvas grades area. The adjustment factor is a value calculated by averaging the students’ scores across all question categories and dividing this by the average of the overall rating each team member received. It can be used as a percentage if multiplied by a hundred. 

How is the Adjustment Factor calculated? 

An Adjustment Factor value is a calculated number that attempts to show the contribution of a given student relative to the other members of their team. Specifically, this Adjustment Factor is the average rating of the student divided by the overall average rating for all members of the team.  

This video is from a previous version of Buddycheck but describes in detail how the adjustment factor is calculated.

With or without self 

With Self means that the student's self-ratings factored in to the adjustment factor. In some cases, exceptional conditions such as 'Cliques' may have an impact on the validity of the Adjustment Factor values, and in these cases, the Adjustment Factor columns will be highlighted.  

Original or capped 

You have the ability to choose between the original or capped adjustment factor. By default, the capped adjustment Factor value is capped at a maximum value of 1.05. Also, values above 0.95 are rounded up to 1.00 (our findings are that values in this range are just 'noise'). You can get an explanation of the capped adjustment factor by hovering over the switch on top of the results table. 

What if a student hasn’t completed an evaluation? 

If a student does not complete the evaluation, there is less peer evaluation data in the group of this student. This means that the results will be less reliable especially if it is a small group or if there are more students in the same group that have not completed the evaluation. That being said, an average and an adjustment factor of a student is simply calculated using completed evaluation data from the peers in the group. So, you can just use all results even if 100% of students have not completed the evaluation. Students that have not submitted do get an average and an adjustment factor because peers probably did complete an evaluation with this person included. 

Working with the results 

In the results tab, you might see Notes appear such as 'High' or 'Low' or 'Under'. On the top of the page, you can find for information on each label. As an instructor, you can use the results to give credits to high performers and talk to low performers or teams with problems. You can use the Adjustment factors to update the course grades too. However, be careful not to directly copy all Adjustment factors, since it often requires your human interpretation to correctly value the trustworthiness of the results and the assessments. The Adjustment Factors are visible on the right side of the Results tab. You have two types: Without Self and With Self. Without, Self is the factor without his or her own (self) ratings. With, Self is with the self-ratings included. For example the Adjustment factor is 1.05 and the student has a 7 (out of 10) as a course grade. You multiple 7 with 1.05, which means the final grade would be 7.35

You can amplify the effect of the adjustment on the final score by using weighting.

Weighting is a multiplier of the difference made by the adjustment factor.

The first part of the calculation works out the adjustment factor of the individual student grade.

So, for example, a student is doing well in the team and has 1.2 adjustment factor and a current grade is 70 out of 100 this equals a difference of 14.

70 x 1.2 = 84

84 - 70 = 14 (the difference between the old and new score)

If you set the weighting of this difference to 0.5 it becomes 70 + (14 x 0.5) =  77

if a student is not doing so well in the team and has 0.8 adjustment factor and a current grade is 70 out of 100 this equals a difference of 14.

70 x 0.8 = 56

70 - 56 = -14 (the difference between the old and new score)

If you set the weighting of this difference to 0.5 it becomes 70 + (-14 x 0.5) =  63

What the student sees and the support 

If the results are released to the students, all students receive a link to a page with their results, including tips on how to improve. 

 

 

 

Support 

Online chat support with FAQ is available from within the BuddyCheck interface. When you open BuddyCheck for the first time, you will see a brief explanation to get started. The instructor receives an email 2 days before the start of the evaluation to remind him to check whether the groups and students have been imported correctly.  

Student Guide: Buddycheck

References

1. Dijkstra, J., Latijnhouwers, M., Norbart, A. and Tio, R.A., 2016. Assessing the “I” in group work assessment: State of the art and recommendations for practice. Medical teacher, 38(7), pp.675-682.
2. Natoli, R., Jackling, B. and Seelanatha, L., 2014. The impact of instructor’s group management strategies on students’ attitudes to group work and generic skill development. Pedagogies: An International Journal, 9(2), pp.116-132.
3. Davies, W.M. 2009 Groupwork as a form of assessment: common problems and recommended solutions. Higher Education 58, 563–584 doi:10.1007/s10734-009-9216-y