FLL’s judging focus is on what we do (the Project and Robot) and how we do it (our Core Values). The judging process is designed to provide meaningful feedback through interaction with the judging panel, as well as evaluate and celebrate each team’s accomplishments throughout the season.
Based on the annual Challenge, teams showcase their experience in three judged areas in addition to the score-based Robot Game. Each judged area utilizes a rubric, or evaluation tool, to gather the information that will drive FLL Deliberations, a qualitative, discussion-based process to rank top teams and determine awards.
FLL Judging recognizes extraordinary team achievement through a set of Core Awards that highlight the most important elements of the FLL experience. Required event award structures are determined based on event type, with some additional local flexibility. All official events are required to provide our Champion's Award to recognize those teams that embody the FLL experience by fully embracing our Core Values while achieving excellence and innovation in both the Robot Game and Project.
It is important to note that FLL believes that all teams who complete the Challenge are successful; awards simply represent the special achievement of particular teams. In addition, every child who participates at an FLL Championship receives a season medallion to signify individual contributions and accomplishments.
Beyond awards, one of the most important aspects of FLL Judging is the written feedback provided to all teams at the conclusion of or shortly following an event.
The most important thing for a Judge Advisor to know about an FLL tournament is that it is supposed to be FUN. FIRST’s mission is to get children excited about science and technology. If you experience any special challenges during the day, focus on that goal. Everything else will fall into place.
As the Judge Advisor you set the tone for the entire judging panel. Teams have worked very hard all season, and they may only have a few, high impact moments with the judges. Please treat team accomplishments and their work with respect, and be sure all judges do as well. One negative comment from a judge can have a devastating effect on teams. Make it your primary goal to ensure that the teams know what they did well, and that they have a positive experience showcasing their achievements. These interactions help each team member to understand that the real reward is what the children discover and learn, both from one another, the judges and through their FLL experience.
Working closely with the Tournament Director, each Judge Advisor is responsible for ensuring the overall judging quality for an event. The Judge Advisor does not judge teams, although he/she should have judged FLL events in the past. Judge Advisors provide overall leadership and are responsible for all judging volunteers, event judging outcomes and the overall team experience related to judging. While specifics may vary slightly by region and event size, key Judge Advisor role elements include judge training , guidance for judges and teams on event day, facilitation of Deliberations, and development of the closing ceremony awards script. Importantly, the Judge Advisor is also responsible to ensure that the FLL Global Standards related to Judging and Awards are followed.
A successful experience for teams and judges begins long before event day. While the most visible role of the Judge Advisor is to manage the judges and judging outcomes on the day of the event, he or she must also work closely with the event organizer, volunteer coordinator and others to make advance decisions about schedules, team communications and other factors to ensure all runs smoothly on event day. While they may not be directly responsible for judge recruitment, the Judge Advisor often oversees this important function as well.
It is important that each member of the planning team have a clear understanding of who is responsible for all activities and decisions. For additional background on some of the key questions, decisions and activities required in advance of an event, please review additional Pre-Event Logistics.
Thoughtful and effective judge recruitment and training have tremendous potential to impact event day, possibly more than any other action within the Judge Advisor's control. Experienced judges who return every year, know the event and understand FLL's judging philosophy and process are able to contribute significantly to a smooth running and equitable deliberations process. They will most definitely make your life easier and the outcomes for the teams better.
Cultivating a strong judging team needs to be viewed as a long term investment. Upfront time to recruit a strong volunteer base is the key first step. Know that even small planning decisions can have a big quality impact, for example pairing experienced and novice judges on the schedule can create a great mentoring opportunity. Similarly, a thank you note to all judges after the event with a "save the date" card can reduce next year's recruiting needs.
Recommendations on the number of required judges as well as a sample description of the judge role can provide additional detail helpful for event planning.
Once the judging team has been recruited, strong training in advance of the event builds consistency and the shared understanding of process that will lead to a smooth and effective deliberations. In addition to ensuring judges understand the requirements and flow of FLL Judging, it is important that you set the stage with the judges so that they recognize the impact their interactions with the children will have. We have captured some key points you may want to review with your judges on the day of the event to remind them of the critical importance of constructive sessions and comments.
FLL has created a number of tools that we hope will be helpful to you as you prepare your local training plan:
Recommended training plan--at a minimum:
Teams should be judged based upon the information provided by them at the event, to you by the tournament organizer and by FLL. Personal opinions that are not based on these materials and the team’s performance should never be part of the judging process. To protect the integrity of the awards, FIRST requires that judges with any affiliation to a team advise other judges of the affiliation, refrain from commenting upon the team, abstain from voting for the team, and refrain from influencing the judges’ decisions on such team in any manner.
While it can be challenging to recruit the significant number of volunteers required to create a successful judge panel, it is important to remember to stretch outside of the immediate FLL community to minimize conflicts of interest. While careful scheduling can help address some concerns, it is often just the perception of a potential conflict that can cause a problem on event day. We recommend use of an appropriate conflict of interest policy (World Festival sample) that is shared with all judges and implemented in advance of the event.
Judging often accounts for the largest number of volunteers at any FLL tournament. As outlined in Pre-Event Logistics, needs will include not only judges, but also a number of other positions that will contribute to a smooth running event day. Depending on event size and set-up, key roles may include:
FLL events fall into five categories. They all offer a fun and exciting way for teams to demonstrate the result of their efforts, but it is important to understand the differences between them:
Community (unofficial local) Events may include demonstrations, practices, scrimmages or summer camps. They are generally, but not always, smaller than official tournaments. They may be run by an FLL Partner or by teams, volunteers, or local coordinators who have the freedom to choose the format, judging guidelines, and awards. Community Events do not qualify a team to attend a Championship tournament and they are not required to follow FLL Global Standards.
Qualifying Tournaments (usually called Qualifiers or Regionals) are the first level of official competition in most regions and must follow FLL Global Standards for both Judging and Awards. While they typically have a similar format to Championship tournaments, they are usually smaller and more low key. Some teams from each Qualifier will be selected to advance to the region’s Championship tournament in accordance with FLL’s Qualifier Advancement policy utilizing Champion’s Award criteria. In some regions there are multiple levels of Qualifiers, sometimes referred to as Super Regionals, that will serve as a team's interim stop on a way to a Championship.
Championship Tournaments are the highest level of competition teams can attend in a state, country or region. Championships must abide by FLL Global Standards. Some Championship tournaments require that teams win at a Qualifying or Regional tournament in order to advance/receive an invitation. For many teams, a Championship tournament will be the highest level of FLL tournament available.
Open Championship Tournaments are invitational events that choose to invite teams from select regions, around the country or the world. Open Championships abide by the same standards as Championships and they are hosted by FLL Partners. These special events usually are not held every year and Partners must apply to FLL Headquarters for approval, typically two years in advance of the event to allow an adequate planning process.
The FLL World Festival is part of the annual FIRST Championship. It is the global celebration of FLL teams from around the world, and it is the only event hosted by FLL Headquarters. World Festival is frequently used as a testing ground for new judging materials and pilot approaches.
All official events are required to provide three separate judged sessions for each team. Each session must be attended by a minimum of two judges, takes place in a separate area and must last a minimum of ten minutes, with additional requirements by area:
Core Values are central to FLL's mission and underlie all elements of our program. Throughout the tournament and particularly during scheduled sessions, Core Values judges will assess how well a team understands and integrates these values into their tournament experience and daily life. In addition to the Rubric, the Core Values Primer provides additional detail and important considerations for area judges.
Core Values judging must take place in a separate session that lasts a minimum of 10 minutes, although we recommend 15 if time allows. Events may utilize one or preferably a combination of several formats:
In addition to session time, Core Values judges will often roam the pits and competition fields to gain additional insights. At some events, separate "floating" judges may be assigned this task as well. Input about a team's Core Values may also come from referees and any other tournament volunteer who should note clear displays of Core Values, or evidence that a team is not practicing them. Occasionally, teams will present to judges one view of FLL Core Values, but on the playing field demonstrate a different understanding. Whether positive or negative, these additional observations can be very helpful during deliberations, especially when teams are closely matched.
FLL has developed a Core Values Input Form to assist in the collection of this valuable supplemental information. As the Judge Advisor, you should ensure that copies of this form are made available to referees and other event volunteers and that an appropriately timed return process is established to allow a review of any additional input prior to finalizing all awards. Note that teams have been advised that an assessment of Core Values may include all team members, and also any coaches, parents, or other individuals associated with the team.
The FLL Challenge requires teams to demonstrate to the judges that all three required annual steps of the Project were completed: they identified a real-world problem, created an innovative solution and shared their findings with others. In addition, they must show that any other Challenge-specific requirements from the season’s Project document were met.
Project interviews take place in a separate judging area, which typically includes a power source and table for presentation purposes. During Project judging sessions, teams are given a total of five minutes to complete their presentation – including setup time. It is expected that the children will be able to handle all set up needs, although appropriate exceptions are allowed.
Teams have been instructed that they will not receive extra time for their session if their audiovisual equipment fails or their presentation runs long. Rather then cut the children off abruptly, most judges will provide a warning and allow them to quickly finish; however, out of fairness to other teams an overly long presentation should be reflected in the rubric assessment and may impact award eligibility in extreme cases.
Teams may perform a skit, a PowerPoint presentation, songs, or choose another creative way to share their Project and communicate their message. Projects must be presented "live", however, with audio/visual tools used only as an enhancement. Some teams may also present a Project binder for review by the judges; note all team materials should be returned to Pit Administration in time for it to be returned to the team before the end of the day.
After the team completes their uninterrupted presentation, judges are provided a question and answer period of at least 5 minutes. Judges may utilize questions found in the Project Judge Prep Pack or other questions developed by the judges if they prefer. In addition to the Rubric, the Project Judging Primer provides important detail and considerations for area judges.
Robot Design is subjectively judged based upon the mechanical design and programming of each team’s robot built for the Robot Game portion of the Challenge. Interviews take place in a separate judging area, which must include an FLL Challenge table (or surface with borders) with a Field Setup Kit.
Robot Design judging typically begins with a team interview initiated by the judges, although some events may choose to utilize the optional Robot Design Executive Summary format being piloted this year. In addition to the Rubric, the Robot Design Judging Primer provides additional detail and important considerations for area judges. Similar to other areas, specific questions to guide the interview session may be found in the Robot Design Judge Prep Pack.
Robot Design judges should have a background in mechanical design and/or programming. Pairing judges with different technical backgrounds can be an effective way to ensure that judges have the required knowledge to choose the winners.
Once all judging sessions are complete, FLL provides a range of awards to recognize and highlight what we believe matters most about the FLL experience. With the exception of the score-based Robot Performance award, all judged awards are determined using a discussion-based ranking process referred to as FLL Deliberations.
Led by the Judge Advisor, FLL Deliberations rely on an in-depth, qualitative review of all teams eligible for awards. Utilizing observations and evaluations captured by the Rubrics as one form of input, judges consider any and all additional team information gathered through call-backs, and informal observations from judges, referees and others. Team achievements are reviewed and contrasted as the judges engage in an often intense discussion to determine which teams will be recognized with awards.
The FLL Deliberations process has a specific flow and incorporates a number of worksheets and tools to help guide each step. Based on event needs and resources, this process may be very low tech or utilize the Judging Lite excel-based application. In a small event with only one Judging Lane, Deliberations will be relatively straight forward. At larger events, the process will be more complex and require more time-- typically a minimum of 90 minutes with additional time to produce award scripts.
Award Descriptions provide more detail about each FLL Award's purpose and selection criteria. Generally, FLL awards fall into three categories:
Official FLL events are required to offer a specific set of awards; it is important that teams share a similar experience no matter where they are in the world. Requirements vary by event type, with some additional local flexibility. High level:
FLL recommends that between 30-50% of all teams at an event receive an award. With limited exceptions, all Core Awards must always be provided to the same tier or depth. When needed, Judges and Local Awards may be utilized to "fill in" event award counts to ensure appropriate levels when a jump to an additional tier would be too much. However, it is important to note that Judges/Local awards may only be used to to recognize teams for an accomplishment not addressed by a Core Award category.
Note that Special Recognition Awards for individuals are strongly recommended, especially at Championships. These include the Outstanding Volunteer, Adult Coach/Mentor and Youth Mentor Awards.
FLL Championship trophies are reserved for Championships. Alternative forms of recognition must be used for Qualifiers and may be used for Special Recognition Awards.
The Champion's Award is required at all official events and it is the most prestigious award that a team can win. A champion is someone who passionately supports a cause, and this award recognizes a team that embodies the FLL experience by fully embracing our Core Values while achieving excellence and innovation in both the Robot Game and Project.
At an official event judges will look for balanced, strong performance across all 3 core areas; this means that all three judged sessions are weighted equally to determine the initial group of Champion’s candidates. All candidates must also meet the following requirements:
All candidate teams are then reviewed during a deliberative process that considers Robot Performance placement and other qualitative factors. Final determination of the award winner(s) is based on a vote of the full judging panel.
Robot Performance is the only score-based area of assessment, and so by definition it is not a judged award. Instead, the Robot Game is officiated on the field by FLL Referees. Referees use the Challenge Mission Descriptions, Rules, Field Setup and Question and Answer materials to guide their scoring.
The Robot Performance Award recognizes the team that scores the most points at an event after having completed at least three 2.5 minute matches--and only the team's highest score counts. If two teams tie, their second highest scores are used to determine which among them has the higher ranking, and so on.
While some events incorporate elimination or alliance rounds for their excitement and fun, these may not be used to determine the Robot Performance Award. Instead, any awards provided may only be Local Awards.
The "R" in FIRST stands for "recognition" and FLL’s award distribution policy is designed to produce an equitable distribution of awards that inspires and celebrates the achievements of as many children as possible. Our policy dictates that no team be given more than one award-- with the exception of Robot Performance, which any team may receive based on points accumulation.
Teams winning awards should be among the highest achievers at a tournament; typically rated exemplary in most rubric categories. However, the objective is not to give every award to the absolute “best” in each category. Instead, the goal is to recognize the entire field of teams in the most appropriate way possible to celebrate the achievements of all teams.
It is important to understand that sometimes our distribution policy means an award may not go to the individual team with the highest ranking in a category if that team is being recognized with another award. Also, note that Special Recognition Awards given to individual coaches and mentors have no impact on eligibility for other team-based awards.
There are a wide range of issues that can impact award eligibility, and as the Judge Advisor, you may need to assess and determine appropriate action when concerns arise at an event. It is important that you take the time you need to feel comfortable that the facts are clear and beyond any reasonable benefit of the doubt. Just like the Head Referee determines the recorded score at the Robot Game table, the Judge Advisor’s word is final when it comes to any judging questions or decisions at an event.
FLL teams work very hard all season and no one likes to see a group of children disappointed, however, it is important that award eligibility be evaluated from the perspective of what is most fair for all teams. Particularly when related to FLL Core Values, an egregious issue in the eyes of the judging team can and should disqualify a team from winning any awards, advancing within the region’s tournament system or participating in any other FLL events for the remainder of the season. Alternately, an exceptional display of coopertition might make the difference in an award determination between two very close teams. FLL has developed a Core Values Input Form that may be distributed and used by all event volunteers (not just judges) to help document Core Values related issues appropriate to share with the Judge Advisor.
When an eligibility issue impacts an otherwise high performing team, it is important that they understand why whenever possible. As the Judge Advisor you may need to tell a coach that the referees reported ungracious behavior and that the team was observed at the practice tables refusing to move when it was another team's turn. Similarly, you may need to relay to a team that because they did not share their project with others before the event they are ineligible for any Project awards despite a fabulous and creative presentation to the judges. Treat these situations as learning opportunities for the team; remember to be factual, respectful, kind, yet also firm when necessary.
The following two eligibility policies are most critical both to understanding FLL fundamentals and to ensuring the fair evaluation of all teams:
Most FLL team members understand the importance of Gracious Professionalism and exhibit it at all times; this is part of what makes FLL and FIRST events so special. We also expect coaches, mentors, parents and other family members to model good behavior and Gracious Professionalism at all times. Additional history and description of the term Gracious Professionalism, including a video that will provide deeper meaning of this hard to define but easy to recognize trait is available and also shared with teams through FLL's website, Coaches Handbook and other means.
For FLL teams, the coach's role is to facilitate the children’s learning, and to support and encourage the team during the season. Adult interference at events-- either during the judging process or robot rounds-- is strictly prohibited.
FLL defines children doing the work as children making all critical decisions in robot building and programming, as well as Project development and presentation. If judges or referees notice adults directing a team’s performance, cuing the team, or prompting children, they may ask the adult to leave the immediate area. Some tournaments choose to restrict the number of adults that accompany children into the judging sessions. These rules are not designed to make the judging or performance process secret, but to ensure fair judging for all with minimal distraction.
In cases where you suspect adult intervention, please be absolutely sure that you have all the information you can. Don’t assume that the children couldn’t do a project or certain programming – ask them! If after more in depth questions to the children, judges believe that adults did the work – or if children tell the judges that their coach or mentor did the work – it would be unfair to other teams to allow award eligibility. Children are usually very honest, and if they can explain why they programmed the robot a certain way, why they chose a certain project topic, or how they arrived at their solution, then the judges will have evidence that the children did the work.
Remember that the starting assumption is that all teams have completed the work on their own, and there must be evidence to the contrary. It is not okay to “feel” like a team had too much coach involvement. Also, please refrain from penalizing a team for being “too prepared.” FLL tournaments showcase these team’s accomplishments, and they naturally want to demonstrate what they have done. Teams will naturally practice for this event, and some may seem less natural than others. Don’t assume that because a team is too polished or prepared that they must not truly understand what they are saying. Again, it is your job to probe and question further to assess their true level of understanding.
This does not mean, however, that judges can go to any length to prove that a team is being less than forthcoming. Judges may not ask for personal information, such as age. Judges may not ask for intellectual property (for example computer programs or research) developed by the children to be stored on their personal media or computers. Teams may elect to provide this information of their own accord, but a process should be in place to ensure the information is returned to the team following the event.
As the Judge Advisor it is important to communicate to all judges that any benefit of the doubt should always fall towards the team. Instruct your judges to involve you in a supplemental interview process if required for all to feel comfortable that the final assessment is the fairest possible.
There are a number of additional considerations that impact award eligibility. As the Judge Advisor, you need to be familiar with these rules and enforce them fairly.
After awards have been determined, the awards script must be prepared for Closing Ceremonies. This is the Judge Advisor’s responsibility, but he or she will require input from the judges who interviewed winning teams. The Awards Script begins with the official FLL Award Descriptions. This text is then supplemented by team-specific details.
For each award, judges will be asked to prepare two to three sentences to explain why the team chosen for that award was picked above all the other teams. This provides a good showcase of the team’s achievements to the public at the Awards Ceremony, and makes the presentation of the award more meaningful for the children and spectators. Judges familiar with the award winning teams should prepare a short explanation describing why that particular team was selected.
The best award scripts incorporate the team name, theme, or something special about the team to provide an element of foreshadowing. In addition, they use pacing and playful humor to engage the audience. Like to see some sample scripts?
Whenever possible, please encourage all judges to attend the Awards Ceremony. Often the Judge Advisor and/or additional judges will be asked to help present awards and to congratulate teams after they receive awards. Many tournaments have a traditional FIRST LEGO League “high five” line for judges to congratulate teams. Don't miss it-- it's a lot of fun and that is what FLL is all about!
It is important to note that the nature and outcome of the deliberations process means that a strict ranking of teams is not possible in judged areas. For example, one team might actually place first across multiple award categories, but through the deliberations process the best award for that team would be determined and the next strongest team would be placed "first" for the additional award.
For this reason and more, the FLL Global Standards for Judging require that no relative information about other teams be made available. Each team should only receive its own rubric-based feedback from judging sessions, and as the Judge Advisor it is important that you review all comments prior to distribution to ensure appropriateness. When time allows, rubrics may be returned at the conclusion of the event or in many cases they are simple scanned/emailed to teams within a short period of time.
In addition, a complete list of results shall be made available to the public to include all officially completed robot rounds and a list of all award winners.
A glossary of terms related to FLL Judging is available and may be helpful review, especially for new Judge Advisors.
© 2011 The United States Foundation for Inspiration and Recognition of Science and Technology (FIRST®) and The LEGO Group. Used by special permission. All rights reserved.