Leaderboard
Last Updated: 09 May 2019, 13:00 (UTC+8)
Note: This was the last update of the leader board. The results below are based on all planner submissions and resubmissions received before the end of 11 April 2019. The final results, which are based on all planners (including planner submissions and resubmissions received on 12 April 2018) and evaluatated on training set and testing set will be announced at the 2019 ICAPS Conference.
Results for Leaderboard
Contributions to the actual portfolio selector
| Rank | Planner | rel. marginal contribution | abs. marginal contribution | Note |
|---|---|---|---|---|
| 1 | kronk | 78.5862% | 0.831332 | |
| 2 | dual-bfws | 7.8717% | 0.083271 | |
| 3 | sysu-planner | 4.5097% | 0.047706 | |
| 4 | IPALAMA | 4.0567% | 0.042915 | |
| 5 | pasar | 3.3140% | 0.035057 | |
| 6 | mrw-rpg | 0.9295% | 0.009833 | |
| 7 | Cerberus | 0.7322% | 0.007746 | |
| 8 | aquaplanning | 0% | 0 |
Contributions to the perfect portfolio selector
| Rank | Planner | rel. marginal contribution | abs. marginal contribution | Note |
|---|---|---|---|---|
| 1 | kronk | 86.1867% | 0.832242 | |
| 2 | dual-bfws | 8.1817% | 0.079005 | |
| 3 | sysu-planner | 2.9377% | 0.028368 | |
| 4 | IPALAMA | 2.2002% | 0.021246 | |
| 5 | pasar | 0.2505% | 0.002418 | |
| 6 | mrw-rpg | 0.2405% | 0.002322 | |
| 7 | Cerberus | 0.0027% | 0.000026 | |
| 8 | aquaplanning | 0% | 0 |
Note: Official results will be determined based on relative marginal contribution to the actual portfolio selector on another new testing set.
About Marginal Contribution
Since the primary goal of Sparkle Planning Challenge 2019 is to analyse the contribution of each planner to the real state of the art, the Sparkle challenge utilises the concept of marginal contribution to measure each planner's contribution to the actual portfolio selector and the perfect portfolio selector, also known as Virtual Best Solver (VBS).
Assume that we have a set of planners $S$ and a portfolio $P$ constructed based on $S$. Let $par10(P)$ denote the PAR10 value achieved by leveraging the complementary strenghts of the planners in $P$.
The absolute marginal contribution (amc) for planner $s$ is calculated as: \begin{equation} amc(s)=\left\{ \begin{array}{ccl} log_{10}{\frac{par10(P\verb|\|\{s\})}{par10(P)}} & & par10(P\verb|\|\{s\})> par10(P) \\ & & \\ 0 & & \text{else} \end{array} \right. \end{equation}
The relative marginal contribution (rmc) for planner $s$ is calculated as: \begin{equation} rmc(s)=\frac{amc(s)}{\sum_{s' \in S}{amc(s')}} \end{equation}
Note: Results for Leaderboard are ranked by preferring greater relative marginal contribution.