Starting in 2023, Code.org's CSA curriculum includes level validation for student work. Level validation was incorporated in response to feedback we received from teachers that students needed clearer guidance on whether they accomplished a task in certain programming levels and how to improve their work. This article explains how the validation works, why we added it, and how teachers can use this as a tool in their classrooms.

How does validation work in Java Lab?

When working in Java Lab, students progress through several stages:

1. Investigating and modifying existing code
2. Writing new code
3. Testing individual methods or classes
4. Running the program

On a Java Lab level, the Run button executes the program and allows students to see the result of the code written in the main method, such as the Painter navigating a maze, an animation played in The Theater, or output to the console. The Test button executes the validation to verify that the code the student wrote performs the expected behavior.

The student experience

If validation exist on a level, students must pass each test before the Finish button is enabled. Clicking the Finish button marks the level complete and updates the level indicator to a filled green bubble.

• Students use the Test button after writing their code to check the behavior of classes, methods, or code segments.
• The results and feedback from the tests can help students confirm that the written code performs the expected behavior and provide feedback to help identify errors or clarify misconceptions.
• Students use the Run button to run the program and see the resulting output.
• Level completion will not revert if the student does something after passing the tests that then causes the tests to fail.

Throughout the curriculum, students will see reminders in the Instructions panel to execute the tests on the level and run their code to confirm the results.

Students can skip the tests on a level by choosing a different level from the level progression at the top. However, this means that the Finish button will not be enabled, and the level will not be marked complete. Students may want to skip the tests to revisit the level at a later time or after conducting a code review to receive peer feedback.

Why do Java Lab levels have validation?

To support students

Students need guidance and feedback on achieving learning goals as they investigate and practice new skills and concepts. However, it can be challenging to provide this to each individual student, especially since many programming tasks can have multiple valid solutions. validation are a form of automated assessment that provide immediate feedback to students to help students concentrate their efforts on specific segments of their code and improve their solutions. 

To support teachers

With these automated assessments, teachers can then focus on providing additional support when needed and engaging more meaningfully with students. The test results can help teachers determine whether students have met learning objectives and are able to apply their skills to develop solutions that meet program specifications. If one or more tests fails, the teacher can quickly identify the misconception or error and help the student focus on the specific aspect of their code that needs correction or improvement.

To increase our ability to deliver culturally responsive curriculum

Code.org's Computer Science A (CSA) curriculum hopes to influence attitudes and perceptions toward software engineering by fostering engagement and confidence through culturally responsive strategies, and increasing a sense of belonging through equitable learning opportunities and outcomes.  We see validation as supporting this goal in the following ways:

Supporting CSA Pedagogical Thread #1: Identity and Culture

• Validation as a form of automated assessment help students transition from dependent learners to independent learners and contribute to the development of their academic mindset.
• Validation supports students in implementing and testing solutions. Students are given the time and space to work through problems and improve their solutions while iteratively building their confidence, knowledge, and skills. The feedback provided from these validation help students develop their problem-solving skills while encouraging persistence, effort, and engagement.

Supporting CSA Pedagogical Thread #4: Software Engineering for All

• When students develop programs to solve problems, their solutions are rarely error-free on the first attempt. Programming is a process of "revision and refinement that consists of many levels of development and testing." Software engineers often write validation to check the behavior of smaller segments of code, which makes it easier to quickly identify and correct bugs in programs.
• In the Code.org AP CSA curriculum, students are encouraged to take on the role of software engineers to plan and implement solutions to meaningful problems. validation on Java Lab levels allow students to experience real-world testing tools and strategies and use these to improve their solutions. Through unit testing of their work, students develop an appreciation of software testing while also improving their approach to testing their own programs.

Which levels have validation?

Validation is included on levels that require students to implement a solution to achieve a specific outcome. On these levels, students are given a problem to solve and the expected outcome of the program that their code should achieve. validation on these levels check that the implemented behavior achieves the expected outcome and provide feedback and guidance to students if a test fails to encourage iteration on their solution.

Some levels ask students to debug an existing program by identifying and fixing the bug. Students should use the tests on these levels to check which tests fail and identify which code segments are not producing the expected result. Using these tests, students identify and correct the bug, whether these are syntax or logic errors, so that the individual code segments and overall program produce the expected results. Open-ended programming tasks and projects do not have validation. These levels are intended for students to apply the skills they have learned to develop a program of personal interest, so methods or classes are not defined or specified to students. These types of tasks and projects should instead be assessed using the rubrics provided in the lesson plans.

Example output in the console that students will see after running validation

How do I help students use validation effectively?

Validation does not replace the teacher. When students struggle to meet the requirements for a unit test, teachers can use this feedback to understand possible misconceptions students might have, identify scaffolding strategies, and provide additional support and guidance to help students identify the error in their code.

Teachers can help students use validation effectively by helping them understand how to interpret the test results and use these to identify errors in their code.

Example of using validation to support students

In the following example, students are asked to instantiate a Painter object and then use the move() and turnLeft() methods to navigate the Painter object to the traffic cone. Before writing any code, the student may choose to execute the validation on the level to identify the requirements they need to fulfill. In this case, the student will see two tests:

• MyNeighborhood.java Test > Instantiate a Painter object 
• MyNeighborhood.java Test > Navigate to the traffic cone (x: 2, y: 1) 

These requirements correlate with the tasks outlined in the Instructions panel. Since the student has not yet written any code to fulfill these requirements, they see the status for both tests indicated as FAILED. Additionally, they are given the following feedback:

The feedback displayed lets the student know that the test failed because either the Painter object has not been instantiated in MyNeighborhood.java or the Painter object tried to move off the grid. Since no code has yet been written, the first step in fixing this problem is to instantiate a Painter object.

The student can choose to run the tests again after completing one or more steps to make iterative improvements to their program to fulfill the requirements. After instantiating a Painter object to fulfill the first requirement, the result for the first test is indicated as SUCCEEDED.

Now students can focus on fulfilling the second requirement. In this case, students need to use the move() and turnLeft() methods to navigate the Painter object to the traffic cone. The test feedback informs the student that the Painter object did not reach the traffic cone and to check the number of times the Painter object moved and turned.

Additionally, students will see the expected and actual result in the feedback. In this case, students see that the expected result was true but the actual result was false. This means that the test is checking if the Painter object reached the traffic cone (true) but found that this had not yet happened (false).

The expected result will vary depending on the requirements of the level. For example, if students are asked to write a method that returns a value, then the unit test on the level may compare the result from the student's code to an expected value.