Terminology
We use the following terminology with additional definitions provided in the glossary.
- Judging Pair: FLL Global Judging Standards require a minimum of two judges to participate in any team judging session. We refer to these judges who will assess a team as the judging pair; note that at some events this judging "pair" may actually include an additional judge(s).
- Judging Lane: A judging lane is composed of three judging pairs -- one pair assigned to a Core Values session, one pair assigned to the Project session and one pair assigned to the Robot Design session. A judging lane may physically be in one large room separated by sound/curtain barriers or occupy three separate rooms.
- Judging Session: A set period of time during each tournament for each team to present information to the judges in the Project, Robot Design and Core Values categories. The format could be any combination of the following: formal presentation, a formal or informal question and answer session, or through observation while completing an activity. FLL Global Judging Standards require sessions to be a minimum of 10 minutes each.
Important Advance Decisions
The following are some decisions which should be made before the Tournament Committee tackles site selection, scheduling, supplies or judge recruitment.
- How many judging lanes will you need? This is an important question! FLL recommends twelve teams per judging lane to avoid overwhelming judging pairs and allow time for appropriate team feedback. A maximum of 16 teams per pair is allowed, but reviewing more than 12 teams can make it extremely difficult for judges to differentiate effectively and stay on schedule. View this chart to see recommended number of lanes, judges and head judges based on the number of teams.
- How many judges will you want per room? Two is required. Three is also acceptable, especially when some judges are relatively inexperienced.
- How much time will a team spend with each judging group? A minimum of ten minutes for a team in front of the judges is required with five private minutes for the judges afterward to make notes and mark rubrics. That results in a schedule requirement of a fifteen-minute block per team. If time permits, consider fifteen minutes instead of ten in front of the judges, for a total of 20 minutes per block.
-
By what schedule/in what sequence do you want the teams to be judged? For example, do you want teams to see their three judging pairs over the course of the day or in one block? In what order will teams be judged? FLL recommends having teams see the all three judging pairs sequentially when possible- Project followed by Robot Design followed by Core Values. This method has been used successfully at World Festival; it keeps the schedule on track and tends to be less stressful for the teams
- Will you do a calibration session? This will impact your schedule and needs to be decided well in advance.
- Will you be doing call-backs? This is important for scheduling as it may require a significant time block once initial judging sessions are complete. Many events are small and do not require callbacks, or do not have sufficient time to schedule them.
- How long would you need for deliberations? It is impossible to predict what will happen during every deliberation process-- some will be smooth and fast, others may be intense and difficult when there are no clear winners or teams are closely matched. Regardless, it is important to schedule a certain amount of time for deliberations and manage judges to the schedule (e.g., set time limits for each part of the process). Many factors affect the process such as the level of experience of the judges, the quality of the teams, passion of the judges, experience level and skill of the Judge Advisor to facilitate the process, etc. FLL recommends Deliberations start as early as possible (often when the judges break for lunch) and schedule a minimum of 90 minutes for discussion and voting, with additional time to prepare award scripts.
Site and Space Requirements
Site selection involves a variety of factors and constraints. The technical director, Judge Advisor and field manager should be part of the site evaluation.
- You will need three quiet rooms per judging lane for team judging, each equipped with a table and chairs for the judges.
- The Project judging room will need a power source and typically a table for team presentation purposes.
- The Robot Design judging room will need an FLL Challenge table (or surface with borders) with field set up kit.
- The Core Values judging room needs will vary.
- A separate private room able to comfortably accommodate all judges is needed to serve as the Judge Deliberation Room. Ideally, this room is set up to allow three discussion pods, one for each judging area.
- When considering a site, imagine the traffic flow as it relates to the team judging. For example (specifics will vary based on your type of schedule): a team walks from the Pits and arrives at "Judging Area Check In"; team gets escorted to the first judging room, then the second, then the third; team exits judging area. Take the time to imagine all that activity. Is the site adequate? What kind of signage will be required?
- What is your policy on coaches and parents being allowed in the judging rooms? How will this impact your space requirement? Or perhaps, the site will dictate the policy. When deciding your policy, consider the logistics and time required to move people in and out of the room, and your desire not to sacrifice the judges' time with the teams. Many events simply choose to allow two adults in the room with the team, a coach and a videographer.
- When your Event Manager considers a site, make sure your requirements are known and thoroughly understood.
The Schedule
It is likely that someone besides the Judge Advisor will be creating the overall event schedule. Take the time to sit down with that person, and make sure your judging needs are known and addressed. A bad schedule can negatively impact teams, judges, judging outcomes and the overall tournament. Review the schedule from both a Judge Advisor and a team perspective.
- As the Judge Advisor, make sure to provide the scheduler with the (1) number of judging lanes, (2) amount of time per judging "block" (for example, the team spends 10 minutes with the judges + 5 minutes for the judges to discuss and evaluate = 15 minute block), (3) judging room locations/numbers, and (4) how your teams will be judged (e.g., in succession)? And in what order?
- If you have more than one lane, how will you divide the teams? Randomly? Numerical order? Distribute the top teams? We recommend that each lane sees a balance of veteran and rookie teams, as well as a range of award potential. For example, do not place all qualifying tournament Champions Award winners in the same room.
- What's happening every minute of the day relative to judging? List it out in detail.
- Develop judging pair/room assignments. Be sure to consider:
- Any potential conflicts of interest
- The experience level of each judge
- Provide each judging room with a customized hour-by-hour schedule including the teams they will judge, break times, Deliberations meeting times, etc. See a sample schedule.
Team Judging Process Decisions
Judging sessions are short. Judges want to learn as much as possible about each team and teams need to be prepared to demonstrate their accomplishments effectively. A number of process decisions will impact how judging sessions are structured:
- Will team profile/introduction sheets be used? Will these be collected in advance of the event? How will they be distributed to judges?
- During Core Values judging, will you ask the teams to perform a teamwork activity? If so, who will be designing this task and acquiring needed supplies?
- During Core Values judging, will the Core Values poster be used? Note the poster is an optional tool that FLL Operational Partners may choose to use to initiate the discussion between teams and Judges during the Core Values Judging. How much time will be needed to accommodate the Core Values poster, and possibly the Teamwork Activity?
- During Robot Design judging, will the Robot Design Executive Summary (RDES) be used? Note the RDES is an optional tool that FLL Operational Partners may choose to use to initiate the discussion between teams and Judges during the Robot Design Judging.
- During the Robot Design judging, will you ask the teams to bring printouts of programs?
The answers to the above questions should be communicated clearly to both teams and judges in advance of the event so they know what to expect.
Judge Recruitment
This process should be started four to six months before the tournament. If you have returning judges, consider sending out a "save the date" email with the previous year's thank you. Get the date on people's calendar early!
Additional important success elements to consider:
- Who will be recruiting judges? The Judge Advisor? The Volunteer Coordinator? Depending on the size of the tournament, the Judge Advisor might consider enlisting the help of an assistant to coordinate judge recruitment and communications.
- How many judges will be needed? Click here for a sample chart.
- Consider overbooking counts by 20% because some judges are likely to cancel.
- Provide a general job description to all judges.
- Recruit / identify your Head Judges, if applicable.
- Look for important qualities:
- All Judges should have experience working with children, be positive and inspirational.
- Robot Design Judges should have some programming and design experience.
- Ideally, at least one Project Judge in each group should have a science background related to the Challenge theme.
- Professionals who have a background in organizational behavior, motivation, teamwork, or communications do well as Core Values Judges.
Team Queuing/Queuers
Team judging can be a crazy, chaotic time for everyone. Make sure the schedule is clear and correct. Make sure judging rooms are labeled with signs. If needed, utilize Judge Queuers to escort teams from the pits to where you want them to go.
- On your event site map, have all key areas for team judging clearly labeled, including traffic flow.
- Where should teams go when it's their time for judging? Where will they report? How will they know this?
- Consider use of a "Team Judging Check In" table at the "entrance" to the Team Judging Area.
- Consider use of a "Judging Area Lead" to check teams in and keep things on schedule. Make sure he/she is organized and handles pressure well.
- Teams should be escorted by a volunteer from the check-in area to their first room; and preferably, if you have enough volunteers, have the same volunteer stay with the same team from beginning to end.
Judge Assistants
Especially for large events, Judge Assistants play a vital role in the judging process. In essence, their role consists of two key components:
- support the Judge Advisor to ensure a well organized and smooth running event.
- support judging room logistics to allow all judges to focus their attention on the teams.
Deliberations Preparations
The deliberation process represents the culmination of each events' judging activities. Especially when time is tight, it can be a pressure-filled part of the day. The best advice to the Judge Advisor is to be prepared:
- Participate in and review the full Judging Manual/Quickstart Guide and additional FLL provided trainings and material, including:
- Determine in advance what level of technology you will use to conduct Deliberations: Judging Lite software? a whiteboard? sticky notes? We recommend use of the excel-based Judging Lite software. Note that use requires a computer able to open and manipulate an Excel file, a projector, and a screen/whiteboard.
- If you plan to use Judging Lite, make sure you understand how to use it before the tournament.
- Make sure you have all the tools you need regardless of your method.
- Be sure to download the award script templates which need to be completed prior to Closing Ceremonies.
Trophies
- Review the required award structures.
- Review the FLL Global Standards related to Judging and Awards
- Which optional/additional awards will be presented? Put your awards list in a document for everyone to reference.
- Communicate decisions to the person ordering trophies.
- Double, and even triple, check this with yourself and with the person placing the order.
- Communicate decisions to the person preparing the program book
Supplies
- Compile a list of supplies in writing, and give to the person in charge of procuring supplies.
- Be sure to include door signs for judging rooms, highlighters, pens & pencils, clipboards, stopwatches/timers, Kleenex, name tags and sharpies, dry erase markers, easels along with BIG Post-It notes & markers, steno pads on which judges can make notes, etc.
- How will judges be identified? Judges shirts? Hats?
Food and Beverages
- Inform the Food Manager about the number of judges, and your food and beverage requirements.
- Remember, judges generally cannot leave rooms during judging sessions, so prepare the rooms with enough beverages (water. caffeinated drinks, etc).
Advance Communication to Teams
Don't wait until the last minute to share communications with teams as coaches tend to be on overload right before tournaments. Work with your Tournament Director to be sure coaches know:
- Tournament schedule, including all judging sessions
- Things to bring for judging (Core Values poster, Robot Design programming printouts, etc.)
- Policy about spectators in the judging rooms
- Policy on the Project judging rooms (e.g., "We can only supply power in the Project judging rooms. If you need other items for your presentation, you must bring your own.")
Team Feedback
Providing appropriate judging feedback to teams is required as outlined in the FLL Global Standards for Judging. It is important to provide a team with a copy of their judging rubrics with the judges' evaluations and comments. Consider the logistics of how these will be returned to teams:
- Rubrics need to be reviewed by the Judge Advisor and/or Head Judges in order to ensure all comments are positive and constructive. When needed, re-write the rubric to remove any negative comments.
- When will you give the teams this feedback? At the end of the tournament, or in the days after the tournament?
- Would you prefer to scan and email after the tournament?
- If you choose to return rubrics at the end of the event, where and how will this occur? How will you communicate this to teams?
- If you choose to return the rubrics at the end of the event, judges typically need their rubrics during the deliberations so there will only be a small window of time between the end deliberations and the end of the tournament in which to:
- Make a copy of all rubrics to retain in case of any post-event issues.
- Distribute rubrics, referee sheets and any additional information to teams.
- For teams that gave judges materials (for example, a binder of their robot's programs), will these need to be returned? Where and how?
© 2011 The United States Foundation for Inspiration and Recognition of Science and Technology (FIRST®) and The LEGO Group. Used by special permission. All rights reserved.