RecSys-HR 2026: WorkRB Challenge

The challenge

Do you want to push the limits of large-scale skill intelligence? Or are you just excited to take home a share of the €4500 prize money?

Skill intelligence still isn’t solved. ESCO has more than 13,000 labels, annotation in this space is far from complete, and the binary yes/no labels used by most published skill-extraction benchmarks routinely miss true positives and treat a near-miss the same as pure nonsense. The result: lexical shortcuts and overfitted classifiers can score as well as systems that genuinely understand the skill space, and the field ends up rewarding the wrong behaviour.

This challenge gives you a sharper instrument. We take the popular open-source skill-extraction and skill-normalisation benchmarks (House, Tech, TechWolf, SkillSkape, SkillNorm) and re-annotate them with graded relevance across five interpretable levels, then release the new annotations to the community (see Evaluation for the label definitions). Your task: recommend the right ESCO skills from free-form text, and normalise extracted skills to the right ESCO concepts.

Everything runs through the open-source WorkRB toolbox: it standardises ranking outputs, generates and scores every submission, and is the same evaluation backbone you can keep using for your own work long after this challenge ends.

Anyone can join

The starter repository contains detailed instructions to quickly bring you up to speed with state-of-the-art baselines and the contributions of this challenge. The development environment lowers the barrier to entry, so anyone can start experimenting with their own ideas (though basic familiarity with machine learning helps 😉).

Prizes^*

🥇

€2000

1st place

🥈

€1000

2nd place

🥉

€500

3rd place

🎓

€1000

Best student team

If the best student team finishes in the top 2, the 3rd-place team receives €1000 and the 4th-place team receives €500.

^* As the sponsor is based outside the US, a small withholding tax may be applied to payouts.

Task & datasets

The task is skill extraction and skill normalisation against ESCO, scored with graded relevance through the WorkRB toolkit. The starter repository ships a hello-world training setup, baselines, and a knowledge notebook so you can get to a first submission quickly.

Training

Any open-source dataset is allowed, except for the benchmarks used in the test set (see Rules). Those benchmarks are also publicly available with their original binary annotations, and those public versions are off-limits too: neither their queries nor their annotations may be used anywhere in your system. The starter scripts already include TechWolf’s synthetic ESCO skill sentences. The SOTA notebook lists further useful skill-extraction datasets, but don’t let the starter files limit your imagination. Maybe regularising with non-skill extraction data helps semantic understanding 👀

Validation

Open-source skill-extraction and skill-normalisation validation sets, enriched with graded-relevance annotations, are available now as new tasks inside WorkRB. While the validation phase is closed, you can still generate your validation score locally using WorkRB.

Test

• Skill extraction: House, Tech, TechWolf, and SkillSkape test sets.
• Skill normalisation: ESCO skill normalisation.

The graded-relevance test annotations were kept hidden for the duration of the challenge and are released through WorkRB now that submissions have closed.

These benchmarks also exist publicly in their original, binary-annotated form. Using those public versions in a submitted system was not permitted, in any capacity: the queries were as off-limits as the labels. See the Rules for the full statement.

Evaluation & metrics

The challenge uses a ranking metric, which means your model must be able to return an ordered list of predictions. This does not restrict you to similarity-based solutions: a classifier can return skills ordered by logits or output probabilities.

Metric: nDCG

nDCG normalises $\mathrm{DCG}@k$ by the ideal $\mathrm{DCG}@k$, giving a score in $[0, 1]$. Given the sparse annotations, we decided to define k to be 100.

Relevance levels

Every (query, skill) pair is assigned one of five graded relevance levels. The levels are defined as follows:

Queries & new annotations

Queries are drawn from popular open-source skill-extraction and skill-normalisation benchmarks: House, Tech, TechWolf, SkillSkape, and SkillNorm. We re-annotate them with the graded-relevance levels above and release the new annotations to the community through WorkRB: the validation set is available now, and the test set counterpart is published after the challenge ends.

Final score

The final score is a macro-average of the nDCG@100 scores.

Timeline

How to participate

1. Register on CodaBench. Provide your name, email, and affiliation, and indicate whether you’re a student (this is verified later if you finish in a prize position on the student track). Registration counts as consent to the challenge rules and to the public release of your team name and leaderboard score.
2. Clone the starter repository. It contains the hello-world training setup, the baselines, a knowledge notebook with the context you need to get going, and the exact code to generate submission-ready files. Recommended starting point: git clone and run the example notebook end-to-end.
3. Submit ranking files to CodaBench. Ranking files are produced by the WorkRB package so the format is consistent across participants. Both phases are now closed: the test phase allowed up to 10 submissions per team, and CodaBench was the only place to see your score.

Stuck on something or want to flag an issue? Email recsys-hr-challenge@techwolf.ai.

Rules

Any violation excludes you from both the main and the student prize tracks.

• The training set may be enriched with any open-source dataset, except for the ones used for testing.
• The test benchmarks are strictly off-limits, in any form and at any stage. House, Tech, TechWolf, SkillSkape, and SkillNorm all exist publicly with their original binary annotations. Those public versions may not be used for anything: not for training, fine-tuning, distillation, pre-training, retrieval, few-shot prompting, nearest-neighbour lookup, thresholding, calibration, model selection, or any other part of a submitted system. This covers both the annotations and the queries, whether the labels are binary, graded, or derived. Matching a test query back to its public entry to recover its label is exactly the behaviour this rule forbids. If your system would score differently with those public files absent, it violates this rule.
• Open-source skill-extraction training strategies cannot be exactly reproduced as your submission. Most of them are already on the leaderboard as baselines anyway.
• Submissions during the validation phase are unlimited. Submissions during the test phase are capped at 10 per team.
• The student competition is, surprisingly, only for students. 😉 The top student team will be requested to prove their student status.
• If the best student team finishes in the top 2 overall, the 3rd-place team receives the €1000 prize and the 4th-place team receives €500.
• A team can win at most one prize. If a team qualifies for more than one (for example, a student team places in both the main and student tracks), it receives only the higher-value prize, and the next eligible team moves up to fill the vacated position.
• By registering, participants consent to the public release of their name and leaderboard scores, which may be discussed in the challenge summary paper.
• The organisers have final say on the interpretation and enforcement of these rules, and may modify, add to, or clarify them at any time. Where feasible, participants will be notified of material changes.
• At the end of the challenge, all participants should submit their training and inference code. Push it to a GitHub repository (private is fine) and add warreveys as a collaborator, then email the repository link to recsys-hr-challenge@techwolf.ai. Code is kept confidential and is not judged on quality.

Support & organisers

Challenge-specific questions: recsys-hr-challenge@techwolf.ai
WorkRB toolkit, integrations, research collaborations: workrb@techwolf.ai

Organising community

• Warre Veys · TechWolf, University of Ghent
• Jens-Joris Decorte · TechWolf
• Luis Gasco · Avature