├── .gitignore ├── README.md ├── pom.xml ├── rst_screenshot.png └── src ├── main ├── java │ └── com │ │ └── github │ │ └── guillaumederval │ │ └── javagrading │ │ ├── CustomGradingResult.java │ │ ├── Grade.java │ │ ├── GradeClass.java │ │ ├── GradeFeedback.java │ │ ├── GradeFeedbacks.java │ │ ├── GradingListener.java │ │ ├── GradingRunner.java │ │ ├── GradingRunnerUtils.java │ │ ├── GradingRunnerWithParameters.java │ │ ├── GradingRunnerWithParametersFactory.java │ │ ├── TestSecurityManager.java │ │ ├── TestStatus.java │ │ └── utils │ │ ├── NaturalOrderComparator.java │ │ ├── PermissionPrintStream.java │ │ ├── PermissionStream.java │ │ └── PrintPermission.java └── main.iml └── test └── java ├── ParametersTests.java ├── PermissionTest.java ├── RunTests.java └── StdTests.java /.gitignore: -------------------------------------------------------------------------------- 1 | .idea/ 2 | target/ 3 | .DS_Store 4 | *.iml 5 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # JavaGrading: grading java made simple 2 | 3 | Simply grade student assignments made in Java or anything that runs on the JVM (Scala/Kotlin/Jython/...). 4 | 5 | ```java 6 | @Test 7 | @Grade(value = 5, cpuTimeout=1000) 8 | @GradeFeedback("Are you sure your code is in O(n) ?", onTimeout=true) 9 | @GradeFeedback("Sorry, something is wrong with your algorithm", onFail=true) 10 | void yourtest() { 11 | //a test for the student's code 12 | } 13 | ``` 14 | 15 | Features: 16 | - CPU timeouts on the code 17 | - Jails student code 18 | - No I/O, including stdout/err 19 | - No thread creating by the student, ... 20 | - Most things involving syscalls are forbidden 21 | - specific permissions can be added on specifics tests if needed 22 | - Text/RST reporting 23 | - Custom feedback, both from outside the test (onFail, onTimeout, ...) but also from inside (see below). 24 | 25 | We use this library at [UCLouvain](https://www.uclouvain.be) in the following courses: 26 | - _Data Structures and Algorithms_ (LSINF1121) 27 | - _Computer Science 2_ (LEPL1402) 28 | - _Constraint Programming_ (LING2365) 29 | 30 | This library is best used with an autograder, such as [INGInious](https://github.com/UCL-INGI/INGInious). 31 | 32 | ## Example 33 | 34 | Add the `@Grade` annotation on your JUnit test like this: 35 | ```java 36 | @RunWith(GradingRunner.class) 37 | public class MyTests { 38 | @Test 39 | @Grade(value = 5) 40 | void mytest1() { 41 | //this works 42 | something(); 43 | } 44 | 45 | @Test 46 | @Grade(value = 3) 47 | @GradeFeedback("You forgot to consider this particular case [...]", onFail=true) 48 | void mytest2() { 49 | //this doesn't 50 | somethingelse(); 51 | } 52 | } 53 | ``` 54 | Note that we demonstrate here the usage of the `@GradeFeedback` annotation, that allows to give feedback to the students. 55 | 56 | You can then run the tests using this small boilerplate: 57 | ```java 58 | public class RunTests { 59 | public static void main(String args[]) { 60 | JUnitCore runner = new JUnitCore(); 61 | runner.addListener(new GradingListener(false)); 62 | runner.run(MyTests.class); 63 | } 64 | } 65 | ``` 66 | 67 | This will print the following on the standard output: 68 | ``` 69 | --- GRADE --- 70 | - class MyTests 8/8 71 | mytest1(StdTests) SUCCESS 5/5 72 | ignored(StdTests) FAILED 0/3 73 | You forgot to consider this particular case [...] 74 | TOTAL 5/8 75 | TOTAL WITHOUT IGNORED 5/8 76 | --- END GRADE --- 77 | ``` 78 | 79 | ## Documentation & installation 80 | 81 | Everything needed is located inside the files: 82 | - [Grade.java](https://github.com/GuillaumeDerval/JavaGrading/blob/master/src/main/java/com/github/guillaumederval/javagrading/Grade.java): annotation for grading a test 83 | - [GradeFeedback.java](https://github.com/GuillaumeDerval/JavaGrading/blob/master/src/main/java/com/github/guillaumederval/javagrading/GradeFeedback.java): annotation to add feedback when a test fails/succeeds/timeouts/is ignored 84 | - [GradeClass.java](https://github.com/GuillaumeDerval/JavaGrading/blob/master/src/main/java/com/github/guillaumederval/javagrading/GradeClass.java): annotation to grade all tests from a class, and to give an overall score 85 | - [CustomGradingResult.java](https://github.com/GuillaumeDerval/JavaGrading/blob/master/src/main/java/com/github/guillaumederval/javagrading/CustomGradingResult.java): exception to be thrown by the "teacher" to give custom score/feedback, if needed 86 | 87 | To add it as a dependency of your project, you can add this to your pom.xml in maven: 88 | ```xml 89 | 90 | com.github.guillaumederval 91 | JavaGrading 92 | 0.5.1 93 | 94 | ``` 95 | 96 | If you are not using maven, [search.maven](https://search.maven.org/artifact/com.github.guillaumederval/JavaGrading/0.5.1/jar) probably has the line of code you need. 97 | 98 | 99 | ## Advanced examples 100 | 101 | ### Cpu timeout 102 | It is (strongly) advised when using an autograder (did I already say that [INGInious](https://github.com/UCL-INGI/INGInious) is a very nice one?) 103 | to put a maximum time to run a test: 104 | ```java 105 | @Test 106 | @Grade(value = 5, cpuTimeout=1000) 107 | void yourtest() { 108 | //a test for the student's code 109 | } 110 | ``` 111 | 112 | If the test runs for more than 1000 milliseconds, it will receive a TIMEOUT error and receive a grade of 0/5. 113 | 114 | Note that if you allow the student (via the addition of some permission) to create new threads, the time taken in the new 115 | threads won't be taken into account! 116 | 117 | It is also possible to add a wall-clock-time timeout, via JUnit: 118 | ```java 119 | @Test(timeout=3000) //kills the test after 3000ms in real, wall-clock time 120 | @Grade(value = 5) 121 | void yourtest() { 122 | //a test for the student's code 123 | } 124 | ``` 125 | 126 | **By default, setting a CPU timeout also sets a wall-clock timeout at three times the cpu timeout.** 127 | If you want to override that, set a different value to `@Test(timeout=XXX)`. 128 | 129 | ### Ignored tests 130 | Ignored tests are supported: 131 | ```java 132 | @Test 133 | @Grade(value = 5) 134 | void yourtest() { 135 | Assume.assumeFalse(true); //JUnit function to indicate that the test should be ignored 136 | } 137 | ``` 138 | 139 | ### Custom feedback (outside the test) 140 | Use the `@GradeFeedback` annotation to give feedback about specific type of errors 141 | ```java 142 | @Test 143 | @Grade(value = 5) 144 | @GradeFeedback("Congrats!", onSuccess=True) 145 | @GradeFeedback("Something is wrong", onFail=True) 146 | @GradeFeedback("Too slow!", onTimeout=True) 147 | @GradeFeedback("We chose to ignore this test", onIgnore=True) 148 | void yourtest() { 149 | // 150 | } 151 | ``` 152 | 153 | ### Custom grade and feedback (inside the test) 154 | Throw the exception `CustomGradingResult` to give a custom grading from inside the text. 155 | 156 | In order to avoid that students throw this exception, this feature is disabled by default. You must activate it by 157 | setting `@Grade(custom=true)` and protect yourself your code against evil students that may throw the exception themselves. 158 | 159 | ```java 160 | @Test 161 | @Grade(value = 2, cpuTimeout=1000, custom=true) 162 | void yourtest() { 163 | try { 164 | //code of the student here 165 | } 166 | catch (CustomGradingResult e) { 167 | throw new CustomGradingResult(TestStatus.FAILED, 0, "Well tried, but we are protected against that"); 168 | } 169 | 170 | if(something) { 171 | throw new CustomGradingResult(TestStatus.FAILED, 1, "Sadly, you are not *completely* right."); 172 | } 173 | else if(somethingelse) { 174 | throw new CustomGradingResult(TestStatus.FAILED, 1.5, "Still not there!"); 175 | } 176 | else if(somethingentirelydifferent) { 177 | throw new CustomGradingResult(TestStatus.TIMEOUT, 1.75, "A bit too slow, I'm afraid"); 178 | } 179 | else if(otherthing) { 180 | throw new CustomGradingResult(TestStatus.SUCCESS, 2.5, "Good! Take these 0.5 bonus points with you"); 181 | } 182 | 183 | //by default, if you throw nothing, it's SUCCESS with the maximum grade 184 | } 185 | ``` 186 | 187 | ### RST output 188 | When using an autograder (I may already have told you that [INGInious](https://github.com/UCL-INGI/INGInious) is very nice) 189 | you might want to output something nice (i.e. not text) for the students. JavaGrading can output a nice 190 | RestructuredText table: 191 | 192 | ```java 193 | public class RunTests { 194 | public static void main(String args[]) { 195 | JUnitCore runner = new JUnitCore(); 196 | runner.addListener(new GradingListener(true)); //notice the *true* here 197 | runner.run(MyTests.class); 198 | } 199 | } 200 | ``` 201 | 202 | ![Screenshot of the RST output](https://raw.githubusercontent.com/GuillaumeDerval/JavaGrading/master/rst_screenshot.png "Screenshot of the RST output") 203 | 204 | ### @GradeClass 205 | The `@GradeClass` annotation allows setting a default grade for all test (avoiding to put @Grade everywhere) 206 | and also to give an overall max grade for the whole class. See next example for... an example. 207 | 208 | ### Parameterized tests 209 | JUnit's parameterized tests are also supported: 210 | 211 | ```java 212 | import com.github.guillaumederval.javagrading.Grade; 213 | import com.github.guillaumederval.javagrading.GradeClass; 214 | import com.github.guillaumederval.javagrading.GradingRunnerWithParametersFactory; 215 | import org.junit.Test; 216 | import org.junit.runner.RunWith; 217 | import org.junit.runners.Parameterized; 218 | 219 | import java.util.Arrays; 220 | import java.util.Collection; 221 | 222 | @RunWith(Parameterized.class) 223 | @Parameterized.UseParametersRunnerFactory(GradingRunnerWithParametersFactory.class) 224 | @GradeClass(totalValue = 100) 225 | public class ParametersTests { 226 | @Parameterized.Parameters 227 | public static Collection numbers() { 228 | return Arrays.asList(new Object[][] { 229 | { 1 }, 230 | { 2 }, 231 | { 3 }, 232 | { 4 }, 233 | { 5 } 234 | }); 235 | } 236 | 237 | int param; 238 | public ParametersTests(int param) { 239 | this.param = param; 240 | } 241 | 242 | @Test 243 | @Grade(value = 1) 244 | public void mytest() throws Exception { 245 | if(param % 2 != 0) 246 | throw new Exception("not even"); 247 | } 248 | } 249 | ``` 250 | 251 | Output: 252 | ``` 253 | - class ParametersTests 40/100 254 | mytest[0](ParametersTests) FAILED 0/20 255 | mytest[1](ParametersTests) SUCCESS 20/20 256 | mytest[2](ParametersTests) FAILED 0/20 257 | mytest[3](ParametersTests) SUCCESS 20/20 258 | mytest[4](ParametersTests) FAILED 0/20 259 | ``` 260 | 261 | ### Multiple test classes 262 | 263 | If you have multiple test classes, simply update the main function like this: 264 | 265 | ```java 266 | public class RunTests { 267 | public static void main(String args[]) { 268 | JUnitCore runner = new JUnitCore(); 269 | runner.addListener(new GradingListener(false)); 270 | runner.run(MyTests.class, MyTests2.class, MyOtherTests.class /*, ... */); 271 | } 272 | } 273 | ``` 274 | 275 | ### Custom permissions 276 | 277 | JavaGrading installs a custom [SecurityManager](https://docs.oracle.com/javase/8/docs/api/java/lang/SecurityManager.html) 278 | that forbids the tested code to do anything that it should not do. 279 | 280 | It effectively forbids [a lot of things](https://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html). 281 | JavaGrading adds an additionnal permission to this list, namely `PrintPermission`, that allows the test code to 282 | print things on stdout/stderr. 283 | 284 | You can re-enable some permissions for a specific test if needed, but it does requires some boilerplate: 285 | 286 | ```java 287 | @RunWith(GradingRunner.class) 288 | public class PermissionTest { 289 | @Test 290 | @Grade(value = 5.0, customPermissions = MyPerms1.class) 291 | public void allowPrint() { 292 | System.out.println("I was allowed to print!"); 293 | } 294 | 295 | @Test 296 | @Grade(value = 5.0, customPermissions = MyPerms2.class) 297 | public void allowThread() { 298 | Thread t = new Thread() { 299 | @Override 300 | public void run() { 301 | // nothing 302 | } 303 | }; 304 | t.start(); 305 | } 306 | 307 | /* 308 | NOTE: the class MUST be public AND static (if it is an inner class) for this to work. 309 | => it must have an accessible constructor without args. 310 | */ 311 | public static class MyPerms1 implements Grade.PermissionCollectionFactory { 312 | @Override 313 | public PermissionCollection get() { 314 | Permissions perms = new Permissions(); 315 | perms.add(PrintPermission.instance); 316 | return perms; 317 | } 318 | } 319 | 320 | public static class MyPerms2 implements Grade.PermissionCollectionFactory { 321 | @Override 322 | public PermissionCollection get() { 323 | Permissions perms = new Permissions(); 324 | perms.add(new RuntimePermission("modifyThreadGroup")); 325 | perms.add(new RuntimePermission(("modifyThread"))); 326 | return perms; 327 | } 328 | } 329 | } 330 | ``` 331 | -------------------------------------------------------------------------------- /pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 4.0.0 6 | 7 | com.github.guillaumederval 8 | JavaGrading 9 | 0.5.1 10 | jar 11 | 12 | 13 | 14 | MIT License 15 | http://www.opensource.org/licenses/mit-license.php 16 | 17 | 18 | 19 | ${project.groupId}:${project.artifactId} 20 | Some tools to grade Java programming exercices. 21 | https://github.com/GuillaumeDerval/JavaGrading 22 | 23 | 24 | 25 | Guillaume Derval 26 | guillaume.derval@uclouvain.be 27 | UCLouvain / ICTEAM / INGI 28 | http://www.uclouvain.be 29 | 30 | 31 | 32 | 33 | 34 | scm:git:git://github.com/GuillaumeDerval/JavaGrading.git 35 | scm:git:ssh://github.com:GuillaumeDerval/JavaGrading.git 36 | https://github.com/GuillaumeDerval/JavaGrading 37 | 38 | 39 | 40 | 41 | 42 | org.apache.maven.plugins 43 | maven-compiler-plugin 44 | 45 | 8 46 | 8 47 | 48 | 3.8.0 49 | 50 | 51 | org.apache.maven.plugins 52 | maven-source-plugin 53 | 2.2.1 54 | 55 | 56 | attach-sources 57 | 58 | jar-no-fork 59 | 60 | 61 | 62 | 63 | 64 | org.apache.maven.plugins 65 | maven-javadoc-plugin 66 | 2.9.1 67 | 68 | --source 8 69 | 70 | 71 | 72 | attach-javadocs 73 | 74 | jar 75 | 76 | 77 | 78 | 79 | 80 | org.apache.maven.plugins 81 | maven-gpg-plugin 82 | 1.5 83 | 84 | 85 | sign-artifacts 86 | verify 87 | 88 | sign 89 | 90 | 91 | 92 | 93 | 94 | org.sonatype.plugins 95 | nexus-staging-maven-plugin 96 | 1.6.7 97 | true 98 | 99 | ossrh 100 | https://oss.sonatype.org/ 101 | true 102 | 103 | 104 | 105 | 106 | 107 | UTF-8 108 | 1.8 109 | 110 | 111 | 112 | junit 113 | junit 114 | 4.13.1 115 | 116 | 117 | 118 | 119 | -------------------------------------------------------------------------------- /rst_screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GuillaumeDerval/JavaGrading/fe7aad60aedd163f6922b3c2e3e67c4bd60e3d49/rst_screenshot.png -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/CustomGradingResult.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading; 2 | 3 | import static java.lang.Double.NaN; 4 | 5 | /** 6 | * Allow a test to return a custom feedback 7 | */ 8 | public class CustomGradingResult extends Exception { 9 | public final String feedback; 10 | public final TestStatus status; 11 | public final double grade; 12 | public final Exception origException; 13 | 14 | /** 15 | * @param status test status 16 | * @param grade the grade. must be NaN to avoid defining a custom grade. Always set to NaN when status == IGNORED 17 | * @param feedback a string describing the feedback, or null 18 | * @param origException original exception, or null 19 | */ 20 | public CustomGradingResult(TestStatus status, double grade, String feedback, Exception origException) { 21 | this.feedback = feedback; 22 | this.status = status; 23 | if(status != TestStatus.IGNORED) 24 | this.grade = grade; 25 | else 26 | this.grade = NaN; 27 | this.origException = origException; 28 | } 29 | 30 | public CustomGradingResult(TestStatus status, double grade, String feedback) { 31 | this(status, grade, feedback, null); 32 | } 33 | 34 | public CustomGradingResult(TestStatus status, double grade) { 35 | this(status, grade, null, null); 36 | } 37 | 38 | public CustomGradingResult(TestStatus status, String feedback) { 39 | this(status, NaN, feedback,null); 40 | } 41 | 42 | public CustomGradingResult(TestStatus status, String feedback, Exception origException) { 43 | this(status, NaN, feedback, origException); 44 | } 45 | 46 | public CustomGradingResult(TestStatus status, double grade, Exception origException) { 47 | this(status, grade, null, origException); 48 | } 49 | 50 | public CustomGradingResult(TestStatus status, Exception origException) { 51 | this(status, NaN, null, origException); 52 | } 53 | 54 | public CustomGradingResult(TestStatus status) { 55 | this(status, NaN, null, null); 56 | } 57 | } 58 | -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/Grade.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading; 2 | 3 | import java.lang.annotation.ElementType; 4 | import java.lang.annotation.Retention; 5 | import java.lang.annotation.RetentionPolicy; 6 | import java.lang.annotation.Target; 7 | import java.security.PermissionCollection; 8 | 9 | @Retention(RetentionPolicy.RUNTIME) 10 | @Target({ElementType.METHOD}) 11 | public @interface Grade { 12 | /** 13 | * Value for the test 14 | */ 15 | double value() default 1.0; 16 | 17 | /** 18 | * CPU timeout in ms. Does not kill the submission, but measures it the time is ok afterwards. 19 | * Should be used with @Test(timeout=xxx). If timeout is not set on @Test, the default will be 3*cpuTimeout 20 | */ 21 | long cpuTimeout() default 0L; 22 | 23 | /** 24 | * Output cputime info, allow printing on stdout/stderr 25 | */ 26 | boolean debug() default false; 27 | 28 | /** 29 | * Expects a CustomGradingResult? 30 | * If false, CustomGradingResult will be considered as a standard error 31 | */ 32 | boolean custom() default false; 33 | 34 | /** 35 | * Overrides permissions. Not taken into account if PermissionCollectionFactory.get() returns null. 36 | * 37 | * The class should be instantiable without args. 38 | * 39 | * By default, tests have no particular permissions, i.e. they can't do anything fancy with the JVM. 40 | * 41 | * Note: if you allow modifyThreadGroup/modifyThread and setIO, you may break some components of JavaGrading, 42 | * namely the protection against stdout/stderr usage and the cpu timeout management. Reflection is also a problem, 43 | * and other permissions may allow tests to jailbreak. Use with caution. 44 | */ 45 | Class customPermissions() default NullPermissionCollectionFactory.class; 46 | 47 | interface PermissionCollectionFactory { 48 | PermissionCollection get(); 49 | } 50 | 51 | class NullPermissionCollectionFactory implements PermissionCollectionFactory { 52 | @Override 53 | public PermissionCollection get() { 54 | return null; 55 | } 56 | } 57 | } -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/GradeClass.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading; 2 | 3 | import java.lang.annotation.ElementType; 4 | import java.lang.annotation.Retention; 5 | import java.lang.annotation.RetentionPolicy; 6 | import java.lang.annotation.Target; 7 | 8 | @Retention(RetentionPolicy.RUNTIME) 9 | @Target({ElementType.TYPE}) 10 | public @interface GradeClass { 11 | /** 12 | * The total value attributed to this test class. 13 | * 14 | * If -1, then each test will keep its original value. 15 | * Else, the total for all the tests in this class will be divided by the maximum value and 16 | * multiplied by totalValue. 17 | */ 18 | double totalValue() default -1.0; 19 | 20 | /** 21 | * Default value for each test in the class. 22 | */ 23 | double defaultValue() default 1.0; 24 | 25 | /** 26 | * If set to true, then all tests in the class must be ok to receive the grade 27 | */ 28 | boolean allCorrect() default false; 29 | 30 | /** 31 | * Default CPU timeout. Similar to the value cpuTimeout in @Grade. 32 | */ 33 | long defaultCpuTimeout() default 0L; 34 | } -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/GradeFeedback.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading; 2 | 3 | import java.lang.annotation.*; 4 | 5 | @Retention(RetentionPolicy.RUNTIME) 6 | @Target({ElementType.METHOD}) 7 | @Repeatable(GradeFeedbacks.class) 8 | public @interface GradeFeedback { 9 | /** 10 | * Message. Use $trace to put the trace or $exception to put the exception. 11 | * $trace and $exception must be alone on their line, with possible whitespaces before them (that will be copied) 12 | */ 13 | String message(); 14 | 15 | /** 16 | * By default, show on failure and timeout. 17 | * If you put any of these to true, then it will only be displayed in this case. 18 | */ 19 | boolean onFail() default false; 20 | boolean onTimeout() default false; 21 | boolean onSuccess() default false; 22 | boolean onIgnore() default false; 23 | } -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/GradeFeedbacks.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading; 2 | 3 | import java.lang.annotation.ElementType; 4 | import java.lang.annotation.Retention; 5 | import java.lang.annotation.RetentionPolicy; 6 | import java.lang.annotation.Target; 7 | 8 | @Retention(RetentionPolicy.RUNTIME) 9 | @Target({ElementType.METHOD}) 10 | public @interface GradeFeedbacks { 11 | GradeFeedback[] value(); 12 | } 13 | -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/GradingListener.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading; 2 | 3 | import com.github.guillaumederval.javagrading.utils.NaturalOrderComparator; 4 | import org.junit.runner.Description; 5 | import org.junit.runner.Result; 6 | import org.junit.runner.notification.Failure; 7 | import org.junit.runner.notification.RunListener; 8 | import org.junit.runners.model.TestTimedOutException; 9 | 10 | import java.io.PrintWriter; 11 | import java.io.StringWriter; 12 | import java.text.DecimalFormat; 13 | import java.text.DecimalFormatSymbols; 14 | import java.util.*; 15 | import java.util.function.Consumer; 16 | 17 | import static java.lang.Double.NaN; 18 | 19 | class Format { 20 | private static DecimalFormat df = initDF(); 21 | 22 | private static DecimalFormat initDF() { 23 | DecimalFormatSymbols otherSymbols = new DecimalFormatSymbols(Locale.ENGLISH); 24 | otherSymbols.setDecimalSeparator('.'); 25 | otherSymbols.getGroupingSeparator(); 26 | return new DecimalFormat("0.##", otherSymbols); 27 | } 28 | 29 | static String format(double d) { 30 | return df.format(d); 31 | } 32 | 33 | static String replace(String orig, String toFind, String replaceBy) { 34 | String[] lines = orig.split("\n"); 35 | for(int i = 0; i < lines.length; i++) { 36 | int foundIdx = lines[i].indexOf(toFind); 37 | if(foundIdx != -1) { 38 | lines[i] = prefix(replaceBy, lines[i].substring(0, foundIdx)); 39 | } 40 | } 41 | StringJoiner sj = new StringJoiner("\n"); 42 | for(String s: lines) sj.add(s); 43 | return sj.toString(); 44 | } 45 | 46 | static String prefix(String orig, String prefix) { 47 | String[] lines = orig.split("\n"); 48 | StringJoiner sj = new StringJoiner("\n"); 49 | for(String s: lines) sj.add(prefix + s); 50 | return sj.toString(); 51 | } 52 | 53 | static String csvEscape(String orig) { 54 | orig = orig.replaceAll("\"", "\"\""); 55 | return "\"" + orig + "\""; 56 | } 57 | 58 | static String statusToIcon(TestStatus status) { 59 | switch (status) { 60 | case IGNORED: 61 | return "❓ Ignored"; 62 | case FAILED: 63 | return "❌ **Failed**"; 64 | case SUCCESS: 65 | return "✅️ Success"; 66 | case TIMEOUT: 67 | return "\uD83D\uDD51 **Timeout**"; 68 | } 69 | return "Unknown status"; 70 | } 71 | } 72 | 73 | /** 74 | * Listener that outputs the grades. 75 | */ 76 | public class GradingListener extends RunListener { 77 | private HashMap classes; 78 | private boolean outputAsRST; //if set to false, outputs as text instead. 79 | 80 | /** 81 | * Outputs to RST by default 82 | */ 83 | public GradingListener() { 84 | super(); 85 | outputAsRST = true; 86 | } 87 | 88 | /** 89 | * @param outputAsRST true to output RST, false to output text 90 | */ 91 | public GradingListener(boolean outputAsRST) { 92 | super(); 93 | this.outputAsRST = outputAsRST; 94 | } 95 | 96 | private GradedClass getGradedClassObj(Class cls) { 97 | if(classes.containsKey(cls)) 98 | return classes.get(cls); 99 | 100 | // Check for annotations 101 | double totalValue = -1; 102 | double defaultValue = -1; 103 | boolean allCorrect = false; 104 | 105 | GradeClass gc = (GradeClass) cls.getAnnotation(GradeClass.class); 106 | if(gc != null) { 107 | totalValue = gc.totalValue(); 108 | defaultValue = gc.defaultValue(); 109 | allCorrect = gc.allCorrect(); 110 | } 111 | 112 | GradedClass gco = new GradedClass(cls, totalValue, defaultValue, allCorrect); 113 | classes.put(cls, gco); 114 | return gco; 115 | } 116 | 117 | private boolean shouldBeGraded(Description desc) { 118 | return desc.getAnnotation(Grade.class) != null || desc.getTestClass().getAnnotation(GradeClass.class) != null; 119 | } 120 | 121 | private void addTestResult(Description description, TestStatus status, double customGrade, Throwable possibleException, String customFeedback, boolean isCustom) { 122 | if(!shouldBeGraded(description)) 123 | return; 124 | 125 | GradedClass gc = getGradedClassObj(description.getTestClass()); 126 | 127 | double maxGrade = gc.defaultValue; 128 | 129 | Grade g = description.getAnnotation(Grade.class); 130 | 131 | if(isCustom && !g.custom()) { 132 | System.out.println("WARNING: Received a CustomGradingResult exception while not expecting one."); 133 | System.out.println("If you are trying to solve this exercise: sadly, there is a protection against this ;-)"); 134 | System.out.println("If you are the exercise creator, you probably forgot to put custom=true inside @Grade."); 135 | 136 | status = TestStatus.FAILED; 137 | customGrade = NaN; 138 | possibleException = null; 139 | } 140 | 141 | if(g != null) 142 | maxGrade = g.value(); 143 | 144 | if(maxGrade == -1.0) 145 | return; 146 | 147 | double grade = 0; 148 | if(!isCustom || Double.isNaN(customGrade)) { 149 | if (status == TestStatus.SUCCESS) 150 | grade = maxGrade; 151 | } 152 | else { 153 | grade = customGrade; 154 | } 155 | 156 | gc.add(description, grade, maxGrade, status, possibleException, customFeedback); 157 | } 158 | 159 | private void addTestResult(Description description, CustomGradingResult customGradingResult) { 160 | addTestResult(description, customGradingResult.status, customGradingResult.grade, customGradingResult.origException, customGradingResult.feedback, true); 161 | } 162 | 163 | private void addTestResult(Description description, TestStatus status, Failure possibleFailure) { 164 | addTestResult(description, status, NaN, possibleFailure == null ? null : possibleFailure.getException(), null,false); 165 | } 166 | 167 | public void testRunStarted(Description description) { 168 | classes = new HashMap<>(); 169 | } 170 | 171 | public void testRunFinished(Result result) { 172 | System.out.println("--- GRADE ---"); 173 | double grade = 0; 174 | double gradeWithoutIgnored = 0; 175 | double max = 0; 176 | double maxWithoutIgnored = 0; 177 | ArrayList gcl = new ArrayList<>(classes.values()); 178 | gcl.sort(new NaturalOrderComparator()); 179 | 180 | if(outputAsRST) { 181 | System.out.println(".. csv-table::\n" + 182 | " :header: \"Test\", \"Status\", \"Grade\", \"Comment\"\n" + 183 | " :widths: auto\n"+ 184 | " "); 185 | } 186 | 187 | for(GradedClass c: gcl) { 188 | if(c.getMax(true) != 0) { 189 | c.printStatus(); 190 | grade += c.getGrade(true); 191 | max += c.getMax(true); 192 | gradeWithoutIgnored += c.getGrade(false); 193 | maxWithoutIgnored += c.getMax(false); 194 | } 195 | } 196 | 197 | if(outputAsRST) { 198 | System.out.println(" \"**TOTAL**\",,**"+Format.format(grade)+"/"+Format.format(max)+"**"); 199 | System.out.println(" \"**TOTAL WITHOUT IGNORED**\",,**"+Format.format(gradeWithoutIgnored)+"/"+Format.format(maxWithoutIgnored)+"**"); 200 | System.out.println(); 201 | } 202 | System.out.println("TOTAL "+Format.format(grade)+"/"+Format.format(max)); 203 | System.out.println("TOTAL WITHOUT IGNORED "+Format.format(gradeWithoutIgnored)+"/"+Format.format(maxWithoutIgnored)); 204 | System.out.println("--- END GRADE ---"); 205 | } 206 | 207 | public void testFinished(Description description) { 208 | addTestResult(description, TestStatus.SUCCESS, null); 209 | } 210 | 211 | public void testFailure(Failure failure) { 212 | if(failure.getException() instanceof TestTimedOutException) 213 | addTestResult(failure.getDescription(), TestStatus.TIMEOUT, failure); 214 | else if(failure.getException() instanceof CustomGradingResult) 215 | addTestResult(failure.getDescription(), (CustomGradingResult)failure.getException()); 216 | else 217 | addTestResult(failure.getDescription(), TestStatus.FAILED, failure); 218 | } 219 | 220 | public void testAssumptionFailure(Failure failure) { 221 | addTestResult(failure.getDescription(), TestStatus.IGNORED, null); 222 | } 223 | 224 | public void testIgnored(Description description) { 225 | addTestResult(description, TestStatus.IGNORED, null); 226 | } 227 | 228 | 229 | 230 | 231 | 232 | 233 | class GradedTest { 234 | final double grade; 235 | final double maxGrade; 236 | final TestStatus status; 237 | private final Description desc; 238 | private final Throwable possibleException; 239 | private final String customFeedback; 240 | 241 | GradedTest(Description desc, double grade, double maxGrade, TestStatus status, Throwable possibleException, String customFeedback) { 242 | this.maxGrade = maxGrade; 243 | this.grade = grade; 244 | this.status = status; 245 | this.desc = desc; 246 | this.possibleException = possibleException; 247 | this.customFeedback = customFeedback; 248 | } 249 | 250 | private String toStringText(double ratio) { 251 | StringBuilder out = new StringBuilder(desc.getDisplayName()).append(" ") 252 | .append(status).append(" ") 253 | .append(Format.format(grade*ratio)) 254 | .append("/") 255 | .append(Format.format(maxGrade*ratio)); 256 | processGradeFeedbacks((s) -> out.append("\n").append(Format.prefix(s, "\t"))); 257 | return out.toString(); 258 | } 259 | 260 | private String toRSTCSVTableLine(double ratio) { 261 | StringBuilder out = new StringBuilder(Format.csvEscape("**→** " + desc.getDisplayName())).append(",") 262 | .append(Format.statusToIcon(status)).append(",") 263 | .append(Format.format(grade*ratio)) 264 | .append("/") 265 | .append(Format.format(maxGrade*ratio)); 266 | 267 | ArrayList fts = new ArrayList<>(); 268 | processGradeFeedbacks(fts::add); 269 | if(fts.size() != 0) 270 | out.append(",").append(Format.csvEscape(String.join("\n\n", fts))); 271 | 272 | return out.toString(); 273 | } 274 | 275 | @Override 276 | public String toString() { 277 | return toString(1); 278 | } 279 | 280 | public String toString(double ratio) { 281 | if(outputAsRST) 282 | return toRSTCSVTableLine(ratio); 283 | else 284 | return toStringText(ratio); 285 | } 286 | 287 | private void processGradeFeedbacks(Consumer op) { 288 | //If there is exactly one @GradeFeedback, feedback is not null, and feedbacks is null. 289 | //If there are more than one @GradeFeedback, feedback is null, and feedbacks is not null. 290 | 291 | if(customFeedback == null) { 292 | GradeFeedback feedback = desc.getAnnotation(GradeFeedback.class); 293 | GradeFeedbacks feedbacks = desc.getAnnotation(GradeFeedbacks.class); 294 | if (feedback != null) 295 | if (shouldDisplayFeedback(feedback)) 296 | op.accept(formatFeedback(feedback.message())); 297 | if (feedbacks != null) { 298 | for (GradeFeedback f : feedbacks.value()) { 299 | if (shouldDisplayFeedback(f)) 300 | op.accept(formatFeedback(f.message())); 301 | } 302 | } 303 | } 304 | else 305 | op.accept(customFeedback); 306 | } 307 | 308 | private boolean shouldDisplayFeedback(GradeFeedback f) { 309 | boolean show = !f.onFail() && !f.onIgnore() && !f.onSuccess() && !f.onTimeout() && (status == TestStatus.FAILED || status == TestStatus.TIMEOUT); 310 | show |= f.onSuccess() && status == TestStatus.SUCCESS; 311 | show |= f.onFail() && status == TestStatus.FAILED; 312 | show |= f.onTimeout() && status == TestStatus.TIMEOUT; 313 | show |= f.onIgnore() && status == TestStatus.IGNORED; 314 | return show; 315 | } 316 | 317 | private String formatFeedback(String feedback) { 318 | if(possibleException != null) { 319 | StringWriter stringWriter = new StringWriter(); 320 | PrintWriter writer = new PrintWriter(stringWriter); 321 | possibleException.printStackTrace(writer); 322 | String trace = stringWriter.toString(); 323 | 324 | return Format.replace(Format.replace(feedback, "$trace", trace), "$exception", possibleException.toString()); 325 | } 326 | return feedback; 327 | } 328 | } 329 | 330 | class GradedClass { 331 | private final Class cls; 332 | private final double totalValue; 333 | final double defaultValue; 334 | private final boolean allCorrect; 335 | private final HashMap grades; 336 | 337 | GradedClass(Class cls, double totalValue, double defaultValue, boolean allCorrect) { 338 | this.cls = cls; 339 | this.totalValue = totalValue; 340 | this.defaultValue = defaultValue; 341 | this.allCorrect = allCorrect; 342 | this.grades = new HashMap<>(); 343 | } 344 | 345 | void add(Description desc, double grade, double maxGrade, TestStatus status, Throwable possibleException, String customFeedback) { 346 | if(!grades.containsKey(desc)) 347 | grades.put(desc, new GradedTest(desc, grade, maxGrade, status, possibleException, customFeedback)); 348 | } 349 | 350 | class GradeResult { 351 | double grade; 352 | double maxGrade; 353 | double maxGradeWithoutIgnored; 354 | boolean allIsSuccess; 355 | boolean allIsSuccessOrIgnore; 356 | 357 | GradeResult() { 358 | grade = 0; 359 | maxGrade = 0; 360 | maxGradeWithoutIgnored = 0; 361 | allIsSuccess = allIsSuccessOrIgnore = true; 362 | } 363 | } 364 | 365 | double getPonderationRatio() { 366 | GradeResult gradeResult = getGradeResult(); 367 | 368 | double realMax = gradeResult.maxGrade; 369 | 370 | if(realMax == 0.0) 371 | return 0.0; 372 | 373 | double destRealMax = totalValue; 374 | if(destRealMax == -1.0) 375 | destRealMax = realMax; 376 | 377 | return destRealMax/realMax; 378 | } 379 | 380 | double getMax(boolean includingIgnored) { 381 | GradeResult gradeResult = getGradeResult(); 382 | 383 | double realMax = gradeResult.maxGrade; 384 | 385 | if(realMax == 0.0) 386 | return 0.0; 387 | 388 | double destRealMax = totalValue; 389 | if(destRealMax == -1.0) 390 | destRealMax = realMax; 391 | 392 | if(includingIgnored) 393 | return destRealMax; 394 | 395 | double notIgnoredRatio = gradeResult.maxGradeWithoutIgnored/gradeResult.maxGrade; 396 | return destRealMax*notIgnoredRatio; 397 | } 398 | 399 | double getGrade(boolean includingIgnored) { 400 | GradeResult gradeResult = getGradeResult(); 401 | 402 | double realMax = gradeResult.maxGrade; 403 | 404 | if(realMax == 0.0) 405 | return 0.0; 406 | 407 | if(allCorrect) { 408 | if(includingIgnored && !gradeResult.allIsSuccess) 409 | return 0.0; 410 | if(!includingIgnored && !gradeResult.allIsSuccessOrIgnore) 411 | return 0.0; 412 | } 413 | 414 | double destRealMax = totalValue; 415 | if(destRealMax == -1.0) 416 | destRealMax = realMax; 417 | 418 | return gradeResult.grade * destRealMax / realMax; 419 | } 420 | 421 | private GradeResult getGradeResult() { 422 | GradeResult r = new GradeResult(); 423 | for(GradedTest t: grades.values()) { 424 | r.allIsSuccess &= t.status == TestStatus.SUCCESS; 425 | r.allIsSuccessOrIgnore &= t.status == TestStatus.SUCCESS || t.status == TestStatus.IGNORED; 426 | r.grade += t.grade; 427 | r.maxGrade += t.maxGrade; 428 | if(t.status != TestStatus.IGNORED) 429 | r.maxGradeWithoutIgnored += t.maxGrade; 430 | } 431 | 432 | return r; 433 | } 434 | 435 | private void printStatusText() { 436 | System.out.println("- " + cls.toString() + " " + Format.format(getGrade(true)) + "/" + Format.format(getMax(true))); 437 | 438 | ArrayList gcl = new ArrayList<>(grades.values()); 439 | gcl.sort(new NaturalOrderComparator()); 440 | 441 | for(GradedTest t: gcl) { 442 | System.out.println(Format.prefix(t.toString(getPonderationRatio()), "\t")); 443 | } 444 | } 445 | 446 | private void printStatusRST() { 447 | String out = " " + 448 | Format.csvEscape("**" + cls.toString() + "**") + "," + 449 | ",**" + 450 | Format.format(getGrade(true)) + 451 | "/" + 452 | Format.format(getMax(true)) + 453 | "**"; 454 | System.out.println(out); 455 | 456 | ArrayList gcl = new ArrayList<>(grades.values()); 457 | gcl.sort(new NaturalOrderComparator()); 458 | 459 | for(GradedTest t: gcl) { 460 | System.out.println(Format.prefix(t.toString(getPonderationRatio()), " ")); 461 | } 462 | } 463 | 464 | void printStatus() { 465 | if(outputAsRST) 466 | printStatusRST(); 467 | else 468 | printStatusText(); 469 | } 470 | 471 | @Override 472 | public String toString() { 473 | return cls.toString(); 474 | } 475 | } 476 | } 477 | -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/GradingRunner.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading; 2 | 3 | import org.junit.Test; 4 | import org.junit.internal.runners.statements.FailOnTimeout; 5 | import org.junit.runners.BlockJUnit4ClassRunner; 6 | import org.junit.runners.model.FrameworkMethod; 7 | import org.junit.runners.model.InitializationError; 8 | import org.junit.runners.model.Statement; 9 | 10 | import java.util.concurrent.TimeUnit; 11 | 12 | /** 13 | * Custom runner that handles CPU timeouts and stdout/err. 14 | */ 15 | public class GradingRunner extends BlockJUnit4ClassRunner { 16 | public GradingRunner(Class klass) throws InitializationError { 17 | super(klass); 18 | } 19 | 20 | @Override 21 | protected Statement methodInvoker(FrameworkMethod method, Object test) { 22 | return GradingRunnerUtils.methodInvoker(method, super.methodInvoker(method, test)); 23 | } 24 | 25 | @Override 26 | protected Statement methodBlock(FrameworkMethod method) { 27 | return GradingRunnerUtils.methodBlock(method, super.methodBlock(method)); 28 | } 29 | 30 | @Override 31 | protected Statement withPotentialTimeout(FrameworkMethod method, 32 | Object test, Statement next) { 33 | return GradingRunnerUtils.withPotentialTimeout(method, test, next); 34 | } 35 | } 36 | -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/GradingRunnerUtils.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading; 2 | 3 | import com.github.guillaumederval.javagrading.utils.PrintPermission; 4 | import org.junit.Test; 5 | import org.junit.internal.runners.statements.FailOnTimeout; 6 | import org.junit.runners.model.FrameworkMethod; 7 | import org.junit.runners.model.Statement; 8 | import org.junit.runners.model.TestTimedOutException; 9 | 10 | import java.lang.management.ManagementFactory; 11 | import java.lang.management.ThreadMXBean; 12 | import java.lang.reflect.InvocationTargetException; 13 | import java.security.*; 14 | import java.security.cert.Certificate; 15 | import java.util.concurrent.TimeUnit; 16 | 17 | import static java.util.concurrent.TimeUnit.MILLISECONDS; 18 | 19 | /** 20 | * Functions common to GradingRunners. 21 | * 22 | * Does all the hard work about stdout/stderr and cpu timeouts. 23 | */ 24 | class GradingRunnerUtils { 25 | static Statement methodInvoker(FrameworkMethod method, Statement base) { 26 | return cpu(method, jail(method, base)); 27 | } 28 | 29 | static Statement methodBlock(FrameworkMethod method, Statement base) { 30 | return base; 31 | } 32 | 33 | static Statement withPotentialTimeout(FrameworkMethod method, Object test, Statement next) { 34 | Test annoTest = method.getAnnotation(Test.class); 35 | Grade annoGrade = method.getAnnotation(Grade.class); 36 | GradeClass annoGradeClass = method.getDeclaringClass().getAnnotation(GradeClass.class); 37 | 38 | long timeout = 0; 39 | 40 | if(annoTest != null) 41 | timeout = annoTest.timeout(); 42 | 43 | if(annoGrade != null && timeout == 0 && annoGrade.cpuTimeout() > 0) 44 | timeout = annoGrade.cpuTimeout() * 3; 45 | 46 | if(annoGradeClass != null && timeout == 0 && annoGradeClass.defaultCpuTimeout() > 0) 47 | timeout = annoGradeClass.defaultCpuTimeout() * 3; 48 | 49 | if (timeout <= 0) { 50 | return next; 51 | } 52 | return FailOnTimeout.builder() 53 | .withTimeout(timeout, TimeUnit.MILLISECONDS) 54 | .build(next); 55 | } 56 | 57 | /** 58 | * Add a test that verifies that a given test do not take too much cpu time 59 | */ 60 | private static Statement cpu(final FrameworkMethod method, final Statement base) { 61 | Grade g = method.getAnnotation(Grade.class); 62 | GradeClass gc = method.getDeclaringClass().getAnnotation(GradeClass.class); 63 | 64 | long cpuTimeout = 0; 65 | if(g != null && g.cpuTimeout() > 0) 66 | cpuTimeout = g.cpuTimeout(); 67 | if(gc != null && cpuTimeout == 0 && gc.defaultCpuTimeout() > 0) 68 | cpuTimeout = gc.defaultCpuTimeout(); 69 | 70 | final long cpuTimeoutFinal = cpuTimeout; 71 | final boolean debug = g != null && g.debug(); 72 | 73 | if(cpuTimeoutFinal > 0) { 74 | return new Statement() { 75 | @Override 76 | public void evaluate() throws Throwable { 77 | ThreadMXBean thread = ManagementFactory.getThreadMXBean(); 78 | long start = thread.getCurrentThreadCpuTime(); 79 | base.evaluate(); 80 | long end = thread.getCurrentThreadCpuTime(); 81 | if(debug) 82 | System.out.println("Function "+ method.toString()+ " took " + ((end-start)/1000000L) + "ms"); 83 | if(end-start > cpuTimeoutFinal*1000000L) 84 | throw new TestTimedOutException(cpuTimeoutFinal, MILLISECONDS); 85 | } 86 | }; 87 | } 88 | else 89 | return base; 90 | } 91 | 92 | private static Statement jail(FrameworkMethod method, final Statement base) { 93 | checkSecurity(); 94 | 95 | final Grade g = method.getAnnotation(Grade.class); 96 | 97 | PermissionCollection coll = null; 98 | if(g != null) { 99 | try { 100 | coll = g.customPermissions().getConstructor().newInstance().get(); 101 | } 102 | catch (NoSuchMethodException | IllegalAccessException | InstantiationException | InvocationTargetException ignored) { 103 | //ignored 104 | } 105 | } 106 | 107 | if(coll == null) 108 | coll = new Permissions(); 109 | if(g != null && g.debug()) 110 | coll.add(PrintPermission.instance); 111 | 112 | ProtectionDomain pd = new ProtectionDomain(new CodeSource(null, (Certificate[]) null), coll); 113 | 114 | return new Statement() { 115 | @Override 116 | public void evaluate() throws Throwable { 117 | 118 | Throwable ex = AccessController.doPrivileged(new PrivilegedExceptionAction() { 119 | @Override 120 | public Throwable run() throws Exception { 121 | Throwable ex = null; 122 | try { 123 | base.evaluate(); 124 | } catch (Throwable throwable) { 125 | ex = throwable; 126 | } 127 | return ex; 128 | } 129 | }, new AccessControlContext(new ProtectionDomain[]{pd})); 130 | 131 | if(ex != null) 132 | throw ex; 133 | } 134 | }; 135 | } 136 | 137 | private static void checkSecurity() { 138 | if(!(System.getSecurityManager() instanceof TestSecurityManager)) { 139 | try { 140 | System.setSecurityManager(new TestSecurityManager()); 141 | } 142 | catch (SecurityException e) { 143 | System.out.println("/!\\ WARNING: Cannot set a TestSecurityManager as the security manager. Tests may not be jailed properly."); 144 | } 145 | } 146 | } 147 | } 148 | -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/GradingRunnerWithParameters.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading; 2 | 3 | import org.junit.runners.model.FrameworkMethod; 4 | import org.junit.runners.model.InitializationError; 5 | import org.junit.runners.model.Statement; 6 | import org.junit.runners.parameterized.BlockJUnit4ClassRunnerWithParameters; 7 | import org.junit.runners.parameterized.TestWithParameters; 8 | 9 | class GradingRunnerWithParameters extends BlockJUnit4ClassRunnerWithParameters { 10 | 11 | public GradingRunnerWithParameters(TestWithParameters test) throws InitializationError { 12 | super(test); 13 | } 14 | 15 | @Override 16 | protected Statement methodInvoker(FrameworkMethod method, Object test) { 17 | return GradingRunnerUtils.methodInvoker(method, super.methodInvoker(method, test)); 18 | } 19 | 20 | @Override 21 | protected Statement methodBlock(FrameworkMethod method) { 22 | return GradingRunnerUtils.methodBlock(method, super.methodBlock(method)); 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/GradingRunnerWithParametersFactory.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading; 2 | 3 | import org.junit.runner.Runner; 4 | import org.junit.runners.model.InitializationError; 5 | import org.junit.runners.parameterized.ParametersRunnerFactory; 6 | import org.junit.runners.parameterized.TestWithParameters; 7 | 8 | public class GradingRunnerWithParametersFactory implements 9 | ParametersRunnerFactory { 10 | public Runner createRunnerForTestWithParameters(TestWithParameters test) 11 | throws InitializationError { 12 | return new GradingRunnerWithParameters(test); 13 | } 14 | } -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/TestSecurityManager.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading; 2 | 3 | import com.github.guillaumederval.javagrading.utils.PermissionStream; 4 | 5 | import java.io.PrintStream; 6 | import java.security.*; 7 | 8 | class TestPolicy extends Policy { 9 | @Override 10 | public boolean implies(ProtectionDomain domain, Permission permission) { 11 | return true; 12 | } 13 | } 14 | 15 | /** 16 | * A custom Security Manager, authorizing everything and adding a new Permission for writing to stdout/stderr 17 | * 18 | * it is automatically as the JVM's Security Manager once a test is run with GradingRunner. 19 | */ 20 | public class TestSecurityManager extends SecurityManager { 21 | 22 | private static ThreadGroup rootGroup; 23 | 24 | public TestSecurityManager() { 25 | System.setOut(new PrintStream(new PermissionStream(System.out))); 26 | System.setErr(new PrintStream(new PermissionStream(System.err))); 27 | Policy.setPolicy(new TestPolicy()); 28 | } 29 | 30 | /** 31 | * Hackfix to forbid creating threads in the root group when you have no rights to create threads 32 | */ 33 | @Override 34 | public ThreadGroup getThreadGroup() { 35 | if (rootGroup == null) { 36 | rootGroup = getRootGroup(); 37 | } 38 | return rootGroup; 39 | } 40 | 41 | private static ThreadGroup getRootGroup() { 42 | ThreadGroup root = Thread.currentThread().getThreadGroup(); 43 | while (root.getParent() != null) { 44 | root = root.getParent(); 45 | } 46 | return root; 47 | } 48 | } -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/TestStatus.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading; 2 | 3 | public enum TestStatus { 4 | SUCCESS, FAILED, IGNORED, TIMEOUT 5 | } 6 | -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/utils/NaturalOrderComparator.java: -------------------------------------------------------------------------------- 1 | /* 2 | * The Alphanum Algorithm is an improved sorting algorithm for strings 3 | * containing numbers. Instead of sorting numbers in ASCII order like 4 | * a standard sort, this algorithm sorts numbers in numeric order. 5 | * 6 | * The Alphanum Algorithm is discussed at http://www.DaveKoelle.com 7 | * 8 | * Released under the MIT License - https://opensource.org/licenses/MIT 9 | * 10 | * Copyright 2007-2017 David Koelle 11 | * 12 | * Permission is hereby granted, free of charge, to any person obtaining 13 | * a copy of this software and associated documentation files (the "Software"), 14 | * to deal in the Software without restriction, including without limitation 15 | * the rights to use, copy, modify, merge, publish, distribute, sublicense, 16 | * and/or sell copies of the Software, and to permit persons to whom the 17 | * Software is furnished to do so, subject to the following conditions: 18 | * 19 | * The above copyright notice and this permission notice shall be included 20 | * in all copies or substantial portions of the Software. 21 | * 22 | * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 23 | * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 24 | * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 25 | * IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 26 | * DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 27 | * OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE 28 | * USE OR OTHER DEALINGS IN THE SOFTWARE. 29 | */ 30 | 31 | package com.github.guillaumederval.javagrading.utils; 32 | 33 | import java.util.Arrays; 34 | import java.util.Comparator; 35 | import java.util.List; 36 | import java.util.stream.Collectors; 37 | 38 | /** 39 | * This is an updated version with enhancements made by Daniel Migowski, 40 | * Andre Bogus, and David Koelle. Updated by David Koelle in 2017. 41 | * 42 | * To use this class: 43 | * Use the static "sort" method from the java.util.Collections class: 44 | * Collections.sort(your list, new AlphanumComparator()); 45 | */ 46 | public class NaturalOrderComparator implements Comparator 47 | { 48 | private final boolean isDigit(char ch) 49 | { 50 | return ((ch >= 48) && (ch <= 57)); 51 | } 52 | 53 | /** Length of string is passed in for improved efficiency (only need to calculate it once) **/ 54 | private final String getChunk(String s, int slength, int marker) 55 | { 56 | StringBuilder chunk = new StringBuilder(); 57 | char c = s.charAt(marker); 58 | chunk.append(c); 59 | marker++; 60 | if (isDigit(c)) 61 | { 62 | while (marker < slength) 63 | { 64 | c = s.charAt(marker); 65 | if (!isDigit(c)) 66 | break; 67 | chunk.append(c); 68 | marker++; 69 | } 70 | } else 71 | { 72 | while (marker < slength) 73 | { 74 | c = s.charAt(marker); 75 | if (isDigit(c)) 76 | break; 77 | chunk.append(c); 78 | marker++; 79 | } 80 | } 81 | return chunk.toString(); 82 | } 83 | 84 | public int compare(Object s1o, Object s2o) 85 | { 86 | String s1 = s1o.toString(); 87 | String s2 = s2o.toString(); 88 | 89 | if ((s1 == null) || (s2 == null)) 90 | { 91 | return 0; 92 | } 93 | 94 | int thisMarker = 0; 95 | int thatMarker = 0; 96 | int s1Length = s1.length(); 97 | int s2Length = s2.length(); 98 | 99 | while (thisMarker < s1Length && thatMarker < s2Length) 100 | { 101 | String thisChunk = getChunk(s1, s1Length, thisMarker); 102 | thisMarker += thisChunk.length(); 103 | 104 | String thatChunk = getChunk(s2, s2Length, thatMarker); 105 | thatMarker += thatChunk.length(); 106 | 107 | // If both chunks contain numeric characters, sort them numerically 108 | int result = 0; 109 | if (isDigit(thisChunk.charAt(0)) && isDigit(thatChunk.charAt(0))) 110 | { 111 | // Simple chunk comparison by length. 112 | int thisChunkLength = thisChunk.length(); 113 | result = thisChunkLength - thatChunk.length(); 114 | // If equal, the first different number counts 115 | if (result == 0) 116 | { 117 | for (int i = 0; i < thisChunkLength; i++) 118 | { 119 | result = thisChunk.charAt(i) - thatChunk.charAt(i); 120 | if (result != 0) 121 | { 122 | return result; 123 | } 124 | } 125 | } 126 | } 127 | else 128 | { 129 | result = thisChunk.compareTo(thatChunk); 130 | } 131 | 132 | if (result != 0) 133 | return result; 134 | } 135 | 136 | return s1Length - s2Length; 137 | } 138 | 139 | /** 140 | * Shows an example of how the comparator works. 141 | * Feel free to delete this in your own code! 142 | */ 143 | public static void main(String[] args) { 144 | List values = Arrays.asList("dazzle2", "dazzle10", "dazzle1", "dazzle2.7", "dazzle2.10", "2", "10", "1", "EctoMorph6", "EctoMorph62", "EctoMorph7"); 145 | System.out.println(values.stream().sorted(new NaturalOrderComparator()).collect(Collectors.joining(" "))); 146 | } 147 | } -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/utils/PermissionPrintStream.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading.utils; 2 | 3 | import java.io.PrintStream; 4 | 5 | /** 6 | * Print, or not, depending on the code having PrintPermission or not. 7 | */ 8 | public class PermissionPrintStream extends PrintStream { 9 | PermissionStream ts; 10 | public PermissionPrintStream(PrintStream s) { 11 | super(new PermissionStream(s)); 12 | ts = (PermissionStream)out; 13 | } 14 | } 15 | -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/utils/PermissionStream.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading.utils; 2 | 3 | import java.io.IOException; 4 | import java.io.OutputStream; 5 | import java.io.PrintStream; 6 | 7 | import static java.lang.System.getSecurityManager; 8 | 9 | /** 10 | * An OutputStream that checks if the code has PrintPermission before printing. 11 | */ 12 | public class PermissionStream extends OutputStream { 13 | PrintStream parent; 14 | boolean warned; 15 | boolean isDisabled; 16 | 17 | public PermissionStream(PrintStream parent) { 18 | this.parent = parent; 19 | warned = false; 20 | isDisabled = "disable".equals(System.getenv("JAVAGRADING_OUTPUT")); 21 | } 22 | 23 | @Override 24 | public void write(int b) throws IOException { 25 | if(!check()) 26 | return; 27 | parent.write(b); 28 | } 29 | 30 | @Override 31 | public void write(byte b[]) throws IOException { 32 | if(!check()) 33 | return; 34 | parent.write(b); 35 | } 36 | 37 | @Override 38 | public void write(byte b[], int off, int len) throws IOException { 39 | if(!check()) 40 | return; 41 | parent.write(b, off, len); 42 | } 43 | 44 | @Override 45 | public void flush() throws IOException { 46 | if(!check()) 47 | return; 48 | parent.flush(); 49 | } 50 | 51 | @Override 52 | public void close() throws IOException { 53 | if(!check()) 54 | return; 55 | parent.close(); 56 | } 57 | 58 | private boolean check() { 59 | SecurityManager sm = getSecurityManager(); 60 | if (sm != null) { 61 | try { 62 | sm.checkPermission(PrintPermission.instance); 63 | } 64 | catch (SecurityException e) { 65 | if(!warned) { 66 | warned = true; 67 | parent.println("WARNING:"); 68 | parent.println("You use print/println/write on System.out or System.err."); 69 | parent.println("It slows down your code a lot. Consider removing/commenting these calls."); 70 | parent.println(); 71 | } 72 | return !isDisabled; 73 | } 74 | } 75 | return true; 76 | } 77 | } -------------------------------------------------------------------------------- /src/main/java/com/github/guillaumederval/javagrading/utils/PrintPermission.java: -------------------------------------------------------------------------------- 1 | package com.github.guillaumederval.javagrading.utils; 2 | 3 | import java.security.Permission; 4 | 5 | public class PrintPermission extends Permission { 6 | public PrintPermission() { 7 | super("print"); 8 | } 9 | 10 | @Override 11 | public boolean implies(Permission permission) { 12 | return equals(permission); 13 | } 14 | 15 | @Override 16 | public boolean equals(Object obj) { 17 | return obj instanceof PrintPermission; 18 | } 19 | 20 | @Override 21 | public int hashCode() { 22 | return 568929722; 23 | } 24 | 25 | @Override 26 | public String getActions() { 27 | return null; 28 | } 29 | 30 | public static final PrintPermission instance = new PrintPermission(); 31 | } 32 | -------------------------------------------------------------------------------- /src/main/main.iml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | -------------------------------------------------------------------------------- /src/test/java/ParametersTests.java: -------------------------------------------------------------------------------- 1 | import com.github.guillaumederval.javagrading.Grade; 2 | import com.github.guillaumederval.javagrading.GradeClass; 3 | import com.github.guillaumederval.javagrading.GradingRunnerWithParametersFactory; 4 | import org.junit.Test; 5 | import org.junit.runner.RunWith; 6 | import org.junit.runners.Parameterized; 7 | 8 | import java.util.Arrays; 9 | import java.util.Collection; 10 | 11 | @RunWith(Parameterized.class) 12 | @Parameterized.UseParametersRunnerFactory(GradingRunnerWithParametersFactory.class) 13 | @GradeClass(totalValue = 100) 14 | public class ParametersTests { 15 | @Parameterized.Parameters 16 | public static Collection numbers() { 17 | return Arrays.asList(new Object[][] { 18 | { 1 }, 19 | { 2 }, 20 | { 3 }, 21 | { 4 }, 22 | { 5 } 23 | }); 24 | } 25 | 26 | int param; 27 | public ParametersTests(int param) { 28 | this.param = param; 29 | } 30 | 31 | @Test 32 | @Grade(value = 1) 33 | public void mytest() throws Exception { 34 | if(param % 2 != 0) 35 | throw new Exception("not even"); 36 | } 37 | } 38 | -------------------------------------------------------------------------------- /src/test/java/PermissionTest.java: -------------------------------------------------------------------------------- 1 | import com.github.guillaumederval.javagrading.*; 2 | import com.github.guillaumederval.javagrading.utils.PrintPermission; 3 | import org.junit.Assume; 4 | import org.junit.Test; 5 | import org.junit.runner.RunWith; 6 | 7 | import java.io.IOException; 8 | import java.io.OutputStream; 9 | import java.io.PrintStream; 10 | import java.security.Permission; 11 | import java.security.PermissionCollection; 12 | import java.security.Permissions; 13 | 14 | @RunWith(GradingRunner.class) 15 | @GradeClass(totalValue = 100) 16 | public class PermissionTest { 17 | @Test() 18 | @Grade(value = 5.0, customPermissions = MyPerms1.class) 19 | public void allowPrint() { 20 | System.out.println("I was allowed to print!"); 21 | } 22 | 23 | @Test 24 | @Grade(value = 5.0) 25 | public void allowPrint2() { 26 | System.out.println("I was allowed to print, but this should not happen if JAVAGRADING_OUTPUT=disable!"); 27 | } 28 | 29 | @Test() 30 | @Grade(value = 5.0, customPermissions = MyPerms2.class) 31 | public void allowThread() { 32 | Thread t = new Thread() { 33 | @Override 34 | public void run() { 35 | // nothing 36 | } 37 | }; 38 | t.start(); 39 | } 40 | 41 | /* 42 | NOTE: the class MUST be public AND static (if it is an inner class) for this to work. 43 | Namely, it must have an accessible constructor without args. 44 | */ 45 | public static class MyPerms1 implements Grade.PermissionCollectionFactory { 46 | @Override 47 | public PermissionCollection get() { 48 | Permissions perms = new Permissions(); 49 | perms.add(PrintPermission.instance); 50 | return perms; 51 | } 52 | } 53 | 54 | public static class MyPerms2 implements Grade.PermissionCollectionFactory { 55 | @Override 56 | public PermissionCollection get() { 57 | Permissions perms = new Permissions(); 58 | perms.add(new RuntimePermission("modifyThreadGroup")); 59 | perms.add(new RuntimePermission(("modifyThread"))); 60 | return perms; 61 | } 62 | } 63 | } 64 | 65 | -------------------------------------------------------------------------------- /src/test/java/RunTests.java: -------------------------------------------------------------------------------- 1 | import com.github.guillaumederval.javagrading.GradingListener; 2 | import org.junit.runner.JUnitCore; 3 | 4 | 5 | public class RunTests { 6 | public static void main(String args[]) { 7 | JUnitCore runner = new JUnitCore(); 8 | runner.addListener(new GradingListener(false)); 9 | runner.run(StdTests.class, ParametersTests.class, PermissionTest.class); 10 | } 11 | } -------------------------------------------------------------------------------- /src/test/java/StdTests.java: -------------------------------------------------------------------------------- 1 | import com.github.guillaumederval.javagrading.*; 2 | import org.junit.Assume; 3 | import org.junit.Test; 4 | import org.junit.runner.RunWith; 5 | 6 | import java.io.IOException; 7 | import java.io.OutputStream; 8 | import java.io.PrintStream; 9 | 10 | @RunWith(GradingRunner.class) 11 | @GradeClass(totalValue = 100) 12 | public class StdTests { 13 | @Test(timeout = 3000, expected = SecurityException.class) 14 | @Grade(value = 5.0, cpuTimeout = 1000) 15 | public void attemptChangeIO() { 16 | PrintStream out = System.out; 17 | System.setOut(new PrintStream(new OutputStream() { 18 | @Override 19 | public void write(int b) throws IOException { 20 | 21 | } 22 | })); 23 | System.setOut(out); 24 | } 25 | 26 | @Test(timeout = 500) 27 | @Grade(value = 5.0) 28 | @GradeFeedback(message = "Timeout, sorry!", onTimeout = true) 29 | @GradeFeedback(message = "Failed, sorry!", onFail = true) 30 | public void shouldTimeout() throws Exception { 31 | System.out.println("test"); 32 | try { 33 | Thread.sleep(1000); 34 | } catch (InterruptedException e) { 35 | e.printStackTrace(); 36 | } 37 | } 38 | 39 | @Test 40 | @Grade(value = 10) 41 | public void ignored() { 42 | Assume.assumeNotNull(null, null); 43 | } 44 | 45 | @Test 46 | @Grade(value = 5) 47 | public void shouldFail() throws CustomGradingResult { 48 | throw new CustomGradingResult(TestStatus.SUCCESS); 49 | } 50 | 51 | @Test 52 | @Grade(value = 5, custom = true) 53 | public void shouldSuccessWithHalfPointsAndComment() throws CustomGradingResult { 54 | throw new CustomGradingResult(TestStatus.SUCCESS, 2.5, "More or less half the points"); 55 | } 56 | 57 | @Test 58 | @Grade(value = 5, custom = true) 59 | public void shouldSuccessWithAllPointsAndComment() throws CustomGradingResult { 60 | throw new CustomGradingResult(TestStatus.SUCCESS, "All the points"); 61 | } 62 | 63 | @Test 64 | @Grade(value = 5, custom = true) 65 | public void shouldTimeoutWith0PointsAndComment() throws CustomGradingResult { 66 | Assume.assumeFalse(true); 67 | throw new CustomGradingResult(TestStatus.TIMEOUT, "Zero!"); 68 | } 69 | } --------------------------------------------------------------------------------