- 11
- 13 904
Harrie Oosterhuis
เข้าร่วมเมื่อ 13 มี.ค. 2012
I am an assistant professor at the Data Science Group of the Institute of Computing and Information Sciences (iCIS) of the Radboud University. My research lies on the intersection of machine learning and information retrieval, it primarily concerns learning from user interactions on rankings. In particular, I focus on methods that correct for the effects of interaction biases.
Unbiased Learning to Rank: On Recent Advances in the Foundations and Applications - SIGIR23 / WSDM24
This video is based on three tutorials presented at the SIGIR 2023 conference in Taipei; Fire 2023 in India; and WSDM 2024 in Mexico.
Slides and more info can be found on the tutorial websites:
sites.google.com/view/sigir-2023-tutorial-ultr
sites.google.com/view/fire-2023-ultr-tutorial
sites.google.com/view/wsdm-2024-tutorial-ultr
Authors:
Shashank Gupta, Philipp Hager, Jin Huang, Ali Vardasbi, Harrie Oosterhuis
Abstract:
Since its inception, the field of unbiased learning to rank (ULTR) has remained very active and has seen several impactful advancements in recent years. This tutorial provides both an introduction to the core concepts of the field and an overview of recent advancements in its foundations along with several applications of its methods. The tutorial is divided into four parts: Firstly, we give an overview of the different forms of bias that can be addressed with ULTR methods. Secondly, we present a comprehensive discussion of the latest estimation techniques in the ULTR field. Thirdly, we survey published results of ULTR in real-world applications. Fourthly, we discuss the connection between ULTR and fairness in ranking. We end by briefly reflecting on the future of ULTR research and its applications. This tutorial is intended to benefit both researchers and industry practitioners who are interested in developing new ULTR solutions or utilizing them in real-world applications.
Chapters:
0:00 - Tutorial Opening and Overview
2:30 - Part 1: Introduction
8:58 - Counterfactual Learning to Rank
18:35 - Inverse Propensity Scoring
34:04 - Part 2: Biases in User Interactions
34:48 - Estimating Position Bias
48:55 - Advanced User Models
56:29 - Trust Bias
1:00:58 - Item Selection Bias
1:02:11 - Item Context Biases
1:06:38 - Part 3: Estimation Methods
1:06:56 - Advanced IPS / Affine Correction
1:10:46 - Policy-Aware Estimation
1:16:40 - Intervention-Aware Estimation
1:25:27 - Two Tower Models
1:33:55 - Doubly-Robust Estimation
1:45:26 - Safety in Optimization
1:58:40 - Part 4: Survey Applications
1:59:18 - Real World Results
2:03:51 - Case Study 1: Grid Layouts
2:08:31 - Case Study 2: Beyond Clicks
2:12:24 - Practical Considerations
2:17:24 - Part 5: Unbiased to Fair LTR
2:17:40 - Introduction to Ranking Fairness
2:32:43 - Unbiased LTR and Fairness
2:39:30 - Part 6: Conclusion
2:42:07 - The Bandwagon Effect: Not Statistical Bias
2:49:13 - Limitations of Unbiased LTR Approach
2:59:18 - Future of Unbiased LTR
3:02:09 - Acknowledgements
Slides and more info can be found on the tutorial websites:
sites.google.com/view/sigir-2023-tutorial-ultr
sites.google.com/view/fire-2023-ultr-tutorial
sites.google.com/view/wsdm-2024-tutorial-ultr
Authors:
Shashank Gupta, Philipp Hager, Jin Huang, Ali Vardasbi, Harrie Oosterhuis
Abstract:
Since its inception, the field of unbiased learning to rank (ULTR) has remained very active and has seen several impactful advancements in recent years. This tutorial provides both an introduction to the core concepts of the field and an overview of recent advancements in its foundations along with several applications of its methods. The tutorial is divided into four parts: Firstly, we give an overview of the different forms of bias that can be addressed with ULTR methods. Secondly, we present a comprehensive discussion of the latest estimation techniques in the ULTR field. Thirdly, we survey published results of ULTR in real-world applications. Fourthly, we discuss the connection between ULTR and fairness in ranking. We end by briefly reflecting on the future of ULTR research and its applications. This tutorial is intended to benefit both researchers and industry practitioners who are interested in developing new ULTR solutions or utilizing them in real-world applications.
Chapters:
0:00 - Tutorial Opening and Overview
2:30 - Part 1: Introduction
8:58 - Counterfactual Learning to Rank
18:35 - Inverse Propensity Scoring
34:04 - Part 2: Biases in User Interactions
34:48 - Estimating Position Bias
48:55 - Advanced User Models
56:29 - Trust Bias
1:00:58 - Item Selection Bias
1:02:11 - Item Context Biases
1:06:38 - Part 3: Estimation Methods
1:06:56 - Advanced IPS / Affine Correction
1:10:46 - Policy-Aware Estimation
1:16:40 - Intervention-Aware Estimation
1:25:27 - Two Tower Models
1:33:55 - Doubly-Robust Estimation
1:45:26 - Safety in Optimization
1:58:40 - Part 4: Survey Applications
1:59:18 - Real World Results
2:03:51 - Case Study 1: Grid Layouts
2:08:31 - Case Study 2: Beyond Clicks
2:12:24 - Practical Considerations
2:17:24 - Part 5: Unbiased to Fair LTR
2:17:40 - Introduction to Ranking Fairness
2:32:43 - Unbiased LTR and Fairness
2:39:30 - Part 6: Conclusion
2:42:07 - The Bandwagon Effect: Not Statistical Bias
2:49:13 - Limitations of Unbiased LTR Approach
2:59:18 - Future of Unbiased LTR
3:02:09 - Acknowledgements
มุมมอง: 249
วีดีโอ
Doubly-Robust Estimation for Correcting Position-Bias in Clicks for Unbiased Learning to Rank
มุมมอง 211หลายเดือนก่อน
This is a lecture based on my 2023 TOIS paper: Doubly-Robust Estimation for Correcting Position-Bias in Click Feedback for Unbiased Learning to Rank The lecture gives an overview of general counterfactual estimation methods to deal with selection bias, and explains how this approach can be adapted for the learning-to-rank setting for optimizing search engines and recommendation systems. PDF and...
Learning-to-Rank at the Speed of Sampling: PL Gradients with Minimal Computational Complexity
มุมมอง 2252 ปีที่แล้ว
Presentation of my SIGIR 2022 short paper: Learning-to-Rank at the Speed of Sampling: Plackett-Luce Gradient Estimation With Minimal Computational Complexity PDF and slides are available here: harrieo.github.io//publication/2022-sigir-short Code available here: github.com/HarrieO/2022-SIGIR-plackett-luce Follow me on twitter: HarrieOos Paper abstract: Plackett-Luce gradient estimati...
Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness
มุมมอง 9453 ปีที่แล้ว
Presentation of my SIGIR 2021 full-paper: Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness PDF and slides are available here: harrieo.github.io/publication/2021-plrank Code available here: github.com/HarrieO/2021-SIGIR-plackett-luce Follow me on twitter: HarrieOos Paper abstract: Recent work has proposed stochastic Plackett-Luce (PL) ...
Recent Advances in Unbiased Learning to Rank from Position-Biased Click Feedback
มุมมอง 2.3K3 ปีที่แล้ว
This lecture was originally presented at an internal meeting at Google; to make it available to the public I have also recorded it for TH-cam. Slides are available here: harrieo.github.io/files/slides/2021-advances.pdf This lecture discusses four recent papers in the unbiased learning to rank field: 1) Policy-Aware Unbiased Learning to Rank for Top-k Rankings - harrieo.github.io//publication/20...
Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank - WWW 2021
มุมมอง 1333 ปีที่แล้ว
The Web Conference 2021 (WWW'21) pre-recorded presentation for our full paper: Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank Harrie Oosterhuis and Maarten de Rijke Preprint and slides are available here: harrieo.github.io/publication/2021-genspec Code available here: github.com/HarrieO/2021WWW-GENSPEC Follow us on twitter: HarrieOos and twitt...
Unifying Online and Counterfactual Learning to Rank - WSDM 2021
มุมมอง 6743 ปีที่แล้ว
The WSDM'21 pre-recorded presentation for our full paper: Unifying Online and Counterfactual Learning to Rank A Novel Counterfactual Estimator that Effectively Utilizes Online Interventions Harrie Oosterhuis and Maarten de Rijke PDF, slides and poster are available here: harrieo.github.io//publication/2021-unifying Code available here: github.com/HarrieO/2021wsdm-unifying-LTR Follow us on twitt...
Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking - ICTIR 2020
มุมมอง 3064 ปีที่แล้ว
The ICTIR'20 pre-recorded presentation for our full paper: Taking the Counterfactual Online: Efficient and Unbiased Online Evaluation for Ranking Harrie Oosterhuis and Maarten de Rijke PDF and slides are available here: harrieo.github.io//publication/2020-logopt Follow us on twitter: HarrieOos and mdr Paper abstract: Counterfactual evaluation can estimate Click-Through-R...
Policy-Aware Unbiased Learning to Rank for Top-k Rankings - SIGIR 2020 Presentation
มุมมอง 1.2K4 ปีที่แล้ว
The SIGIR'20 pre-recorded presentation for our full paper: Policy-Aware Unbiased Learning to Rank for Top-k Rankings Harrie Oosterhuis and Maarten de Rijke Preprint and slides are available here: harrieo.github.io//publication/2020-sigir-topk Follow us on twitter: HarrieOos and mdr Paper abstract: Counterfactual Learning to Rank (LTR) methods optimize ranking systems usi...
Introduction to Counterfactual Learning to Rank - Talk at Farfetch
มุมมอง 1.1K4 ปีที่แล้ว
This talk was originally presented (remotely) at Farfetch on June 23, 2020, it provides an introduction to Counterfactual Learning to Rank from User Interactions. Slides are available here: harrieo.github.io/files/slides/2020-farfetch.pdf This talk is based on this (much longer and advanced) WWW'20 tutorial: th-cam.com/video/BEEfMrn9T9/w-d-xo.html In contrast, this shorter talk is meant for peo...
Unbiased Learning to Rank: Counterfactual and Online Approaches - The Web Conference 2020 Tutorial
มุมมอง 6K4 ปีที่แล้ว
This tutorial video was made for the Web Conference 2020. Originally it would have been presented in Taipei, Taiwan, but due to the COVID-19 pandemic it was recorded remotely. Slides and more info on our website: ilps.github.io/webconf2020-tutorial-unbiased-ltr/ Title: Unbiased Learning to Rank: Counterfactual and Online Approaches Authors: Harrie Oosterhuis, Rolf Jagerman and Maarten de Rijke ...
Thanks for your amazing works and explanations! Can you also make a video of your new Doubly Robust LTR paper? I find it really interesting!
Thank you! I hope to find time for it in the next month.
awesome talk!
Thank you professor, this is a nice talk. Could you share some video or material about how to estimate the bias which was skipped in this talk?
Great talk @Harrie Oosterhuis. One quick question. How do you calculate the the P(o_i = 1 / R, d_i) from the logging data? are we just using some historical data about the "golden triangle". In other words, in real world data it seems difficult to know if an item was observed when there are no clicks below it
Never mind I see it is covered later in the talk. Randomization seems a bit impractical though. Any results from using the "golden triangle" from User research to calculate P(o_i = 1 / i)?
Thanks Harrie! Really interesting talk!
Thank you for sharing with us this great talk. It would be so great if other researchers also shared theirs talks.
One question- does Learning to Rank imply that listwise approach is used? If this is the case, is it true that counterfactual learning to rank, IPS weighting apply only to the listwise approach?
Thank you for the question! LTR does not imply a listwise approach, though generally listwise losses are the best choice. You can also apply counterfactual LTR to pointwise or pairwise losses, we actually discuss a pairwise loss at 58:10
On the slide 14 you have a formula for LambdaRank loss. As far as I understand LambdaRank, this formula is incorrect: closed form for LambdaRank is unknown and DeltaNDCG factor should be added not at the stage of loss calculation, but plugged in directly to the gradient. Essentially they replace gradients (which don't exist for NDCG) with lambdas, and use these lambdas instead of gradients.
Good point, you are correct that the true loss is unknown. We forgot to explain that for this formula the |Delta NDCG| factor should be treated as a constant when calculating the gradient for a pair, then this does lead to the LambdaRank gradient. This idea of treating it as a constant comes from the paper: "The LambdaLoss Framework for Ranking Metric Optimization" where they describe the LambdaRank method as an Expectation Maximization procedure. I hope that clears up the confusion.
@@harrieo Thanks for clarification. You are right, if treat |Delta NDCG| as a constant, that should work.
Great explanation thanks! Would help if you could define what the different logging policies do, is it that we allow the ranking model to change while running an online experiment?
Yes I think you understood it: an intervention means the logging policy is replaced at that moment during the logging of data. So after the moment of intervention the new logging policy decides which rankings to show to users, thereby changing the data that will be logged from thereon. I can't define what the logging policies do exactly since their behavior is learned, we simply took what the optimization thought was the best logging policy (based on the data gathered so far) and intervened to replace the old policy with this new policy. This is similar to a pure exploitation strategy in reinforcement learning.
Thank you so much for sharing the great tutorial. It's really helpful to understand unbiased LTR.
Amazing tutorial!
Great thanks for both of your speaking for unbiased learning. I have learned a lot from your contributions, which is exactly what I cannot gain from reading many papers on prominent conference paper.
This is very helpful :)