To facilitate deliberations at an FLL event requires preparation, a strong understanding of process steps, and an ability to be both flexible and firm. Deliberations are more an art than a science, and expect that each experience will be different. Some sessions will flow quickly and smoothly with all judges reaching an easy agreement on clear award winners; other times when teams are well-matched, judges less experienced or tough-minded, the process will require strong leadership from the Judge Advisor to keep the discussions productive within tight time constraints.
While no two deliberative sessions will ever be exactly alike, the FLL Deliberations process (see flowchart) is intentionally designed to run the same way for all events. Head Judges should be used in each area to assist with facilitation as needed; even if event size does not warrant separate Head Judge positions, an individual should be tapped in each area to take on a leadership role. In addition, we recommend a review of the Deliberations Room Covenants with the full judging group before the start of the day.
Events may be large or small; they may be facilitated with Judging Lite software or sticky notes. But the Judge Advisor should always be comfortable relying on the same key process steps outlined below.
The first step in the judging process is for each team to meet with and be evaluated by a group of at least two judges for a minimum of 10 minutes during three structured sessions. Judging Pairs from each of of the three areas (Core Values, Project and Robot Design) gather the data and information that will be used to determine all awards. The integrity of the awards is contingent upon the judges being able to document and explain why a team earned a coveted award. In addition, judging sessions provide the positive and constructive interaction and feedback teams crave after a season of hard work.
Session time is extremely short and fast-paced. To ensure teams are evaluated in a fair and consistent manner, each Judging Pair must complete one FLL-provided Rubric that will drive the deliberative process and ultimately be returned to each team. Judges may also choose to maintain a private notepad with additional comments as a memory aid. Most events allow at least 5 private minutes at the end of the team session with additional time provided during breaks and other opportunities for judges to complete the rubric as accurately and fully as possible. Note that all teams are provided advance access to the Rubric templates on the FLL website and they are encouraged to use them as a road map to chart their achievements throughout the season.
Judging Pairs are free to determine the technique that works best for them to capture information on the rubric during the session. Some will divide the rubric into sections and/or split the interviewer role. Others will have one person interview while the second serves as scribe. Some prefer to both ask questions and complete the full rubric together. Any method is fine as long as it is effective for the pair.
It is important to note that only ONE rubric should be completed for each team. While some judges may prefer to complete individual rubrics (and this is fine as an interim step), the process requires the Judging Pair to provide the Judge Advisor with one complete rubric that consolidates all comments and fairly reflects both viewpoints. This is the first step in building the consensus of all judges required to determine award winners.
Critical Outcome: Completed Rubrics and comments: one per room
After all team sessions and Rubrics are completed, the judging process shifts from evaluating the achievement of individual teams to selecting the best achievements to recognize with awards from among all the teams. When there is only one Judging Pair assigned to each area this is a relatively easy process as all Judges have seen all teams. For larger events, a series of additional steps are required to merge the findings and recommendations of all Judging Pairs.
Each Judging Pair initially reviews their teams seen throughout the day and prepares a Nomination Worksheet specific to their judging area (see Project Nomination Worksheet, Robot Design Nomination Worksheet, Core Values Nomination Worksheet).
The purpose of the worksheet is to capture:
Once complete all Nomination Worksheets should be provided to the Judge Advisor for data entry into Judging Lite and/or use in subsequent process steps.
Critical Outcome: Individual Award Nominations- approximately 2 per room
Critical Outcome: 1-N Ranking for overall area- one per room (N= the number of teams reviewed by the Judging Pair)
Once the Judge Advisor and Head Judges have reviewed the Nomination Worksheets, they are able to look across all three judged areas to identify potential Champion's Award Candidates. There is no one formula to determine the list of teams to be considered. The appropriate number of candidates will also vary dependent on tournament size-- typically 5-6 is a reasonable award candidate pool.
As a starting point, consider:
To ultimately be selected, a Champion's Award winner also must meet several additional criteria in addition to strong performance BALANCED across all three judged areas; if there are any known issues at this early stage it is wise to adjust the list. All Champion's Award winners must:
As a final check before finalizing the list of potential candidates, the Judge Advisor should always review the list of candidates with the full judging group to make sure no teams are either missing that should not be on the list, or on the list but should not be.
Critical Outcome: Candidates group selected by JA and HJ and confirmed by all judges.
If time for call-backs has been built into the overall event schedule, the Head Judges working closely the the Judge Advisor next determine and communicate assigned call-back times.
Any or all Champion's Award candidates may be added to the call-back schedule as necessary so that more judges may interact with and observe these teams. In addition, any Judging Pair can ask to have a team called back for reasons identified on the nomination worksheets. Reasons may range from an interruption or translation concern to the simple desire to hear more from a team that is in close contention for a top award.
The purpose of call-backs is to gather any additional information about teams that is necessary for judges to make decisions. Call-backs may be formal and take place at a set time in a judging room. Some events prefer to schedule call-backs so that judges across multiple areas can see the same team at the same time. Call-backs may also be informal where judges simply revisit teams in the pits or observe them on the field. During these times judges need to be sensitive to other activities that may require a team's focus, e.g. repairing their robot in the pits, or reviewing strategy for an upcoming round.
Note that it is very easy for the call-back schedule to create false expectations with teams and it is important to communicate to teams that a call-back does not imply they will win an award; similarly a lack of a call-back should not generate an expectation that a team won't win an award. Typically call-backs are occurring during final Robot Game rounds, so this can be a hectic and intense time for teams.
At some event, including World Festival, judges were actually assigned to ensure ALL teams (including those not nominated for any award) received at least a quick visit from judges during the time blocked for scheduled call-backs. We called them our "Ambassador Judges", and this approach was well received by both teams and judges alike.
The next step in the deliberative process is for all Judges in each core area to meet together to discuss all teams nominated for awards. The purpose of this time is to begin the process of developing a ranking of teams by award that reflects the consensus of all judges within the area. It is helpful if the deliberations room can be set-up or re-arranged to allow each of the three groups to meet in a separate pod for focused discussion.
These important discussions are facilitated by area Head Judges who may be a separate individual or an active judge who has been asked to fill this role at a smaller event (see Head Judge Recommendations). Even for a relatively small event it is important to have one person charged with managing the overall discussion. It is easy for Judging Pair presentations to become passionate and exceed time limitations.
Generally one individual from each Judging Pair will serve as spokesperson to highlight key reasons why teams were nominated for particular awards. Specifics can be very important and persuasive as relative strengths and weaknesses are assessed; good notes from interviews can be particularly helpful as the details provided for discussion serve to calibrate both the qualities of the teams and the expectation level of the judges. While the rubrics are designed to minimize personal differences, judging is by its nature subjective and opinions/issues will rarely be black and white.
To be fair to all teams and judges, all presentations should be kept short and factual. To preserve an appropriate balance, we recommend that a 2 minute (+/- as time allows) limit be set for each presentation and strictly enforced.
Some judging groups have each Judging Pair present all teams they nominate for a particular award at one time to be most time efficient; others may prefer to have each Pair's number one nominee presented before circling around to hear each Pair's second nominee and so on. Whatever approach works best for the group is fine.
Note that any judge with a conflict of interest must abstain from participating, or providing any input during discussions (and any subsequent vote).
Once all nominated teams have been presented to all area judges, the group must work together to rank order these teams in a fair manner. The outcome from this step in the process should be captured on the appropriate area ranking worksheet: Project Ranking Worksheet, Core Values Ranking Worksheet, Robot Design Ranking Worksheet. Note that worksheets will capture both the merged ranking for the three area Core Awards, as well as additional collective area rankings for Judges Awards, Adult Coach/Mentor or Youth Mentor awards, if any.
After presentation and discussion of candidates, voting is typically the easiest method to create a merged ranking of all nominated teams. For example, at World Festival the Project judges chose to use the following process:
In the case of a tie, every judge re-voted with one vote. During tie-breaking votes, a judge could vote for a team they had judged.
Process was repeated for each award.
Some regions prefer to use an expanded ranking process. For example, when trying to create a merged ranking for 14 teams, the ranking slots may actually range from 1-100 to allow more "space" to distinguish between teams.
While the process used may vary, the end result is the same-- for each award, a merged ranking of all teams representing the consensus opinion of the judging area is created. Once complete all Ranking Worksheets should be provided to the Judge Advisor for data entry into Judging Lite and/or use in subsequent process steps.
Critical Outcome: 1-N ranking for each award in each area (N= the number of teams nominated for each award)
After all judges in each category complete their rankings for each award, it is time for Final Deliberations to choose and finalize the award winners. The first award to be determined is always the Champion’s Award, as it is FLL's most prestigious award.
Now with all judges from all three areas participating together, the Judge Advisor organizes and leads this important discussion. For each team identified as a potential Champion's candidate (from the list created during step 3 of the deliberations process):
One representative from each of the three core areas provides a short, factual, timed presentation to highlight key team strengths for all the judges
For example, 2 minutes from Core Values, 2 minutes from Project and 2 minutes from the Robot Design judge representative per candidate team;
Any other pertinent information is disclosed by the Judge Advisor to the group, for example
Core Values inputs from other volunteers-- may be exceptionally positive or negative
Robot Performance ranking detail obtained from the fields.
Note that any judge with a conflict of interest must abstain from participating, or providing any input (and any subsequent vote).
After presentations, questions, and sufficient discussion of all teams in consideration, the formal vote for Champion's Award winner(s) takes place.
The teams are rank ordered from most “Yes” votes to fewest.
After the Champion’s Award winners are chosen by the above process, teams that do not win a Champion’s Award are considered for awards under the categories where they were originally nominated.
Awards are assigned based on the rankings previously determined for Core Awards during initial deliberations. If no team is highly placed for multiple awards, this part of the process can flow very quickly. When Judging Lite is used, it is relatively easy to review the award rankings of nominated teams across all areas to determine any potential overlaps.
It is important to remember that with the exception of the objectively determined Robot Performance award, no team is allowed to win more than one team award in accordance with the FLL Awards Distribution Policy.
When a single team is ranked high in more than one Core Award category, the team should be given an award based on the highest ranking it received as all Core Awards (beyond the Champion's Award) are of equal weight. When award placements are in fact equal, however, a determination must be made as to the most appropriate award to give that team. For example:
When all Core Awards have been determined, the Judge Advisor should check with judges to ensure they are comfortable with the choices. Note that not all judges will necessarily agree with all the choices for individual teams, but the judging team as a whole is looking for consensus and a level of comfort with the results. Remember the goal should be to recognize the entire field of teams in the most appropriate way possible to celebrate the achievements of ALL teams.
Once the Core Awards are set, several additional jobs remain:
Critical Outcome: Scripts for all awards
Critical Outcome: Written rubric-based feedback to all teams
© 2011 The United States Foundation for Inspiration and Recognition of Science and Technology (FIRST®) and The LEGO Group. Used by special permission. All rights reserved.