Boseung Jeong

I am a Ph.D. candidate in the Computer Science and Engineering (CSE) department at POSTECH, working in the Computer Vision Lab, advised by Prof. Suha Kwak.

My research primarily focuses on multi-modal retrieval, where I explore how to align and fuse information from different modalities such as vision, language, and audio. I have worked extensively on text-based person search, aiming to retrieve person images using textual attribute queries, and on video-text retrieval, where I study how to represent and align video content with natural language descriptions. I’m particularly interested in developing models that bridge the modality gap and generalize well across real-world data. If you are interested in my research projects, please feel free to contact me.

News

Jun 11, 2025	I am selected as outstanding reviewers in CVPR 2025.
Feb 27, 2025	Our paper on video-text retrieval is accepted to CVPR 2025 (Oral Presentation).
Jul 02, 2024	Two papers are accepted to ECCV 2024.

Education

Sep, 2018 - Present	Pohang University of Science and Technology (POSTECH), Pohang, South Korea Integrated M.S./Ph.D. student in Computer Science and Engineering Advisor: Prof. Suha Kwak
Mar, 2014 - Sep, 2018	Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu, South Korea B.S. in School of Undergraduate Studies

Publications

* indicates equal contribution.

Learning Audio-guided Video Representation with Gated Attention for Video-Text Retrieval

Boseung Jeong, Jicheol Park, Sungyeon Kim, and Suha Kwak

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025

Oral presentation (96/13,088=0.74%)
Abs arXiv

Video-text retrieval, the task of retrieving videos based on a textual query or vice versa, is of paramount importance for video understanding and multimodal information retrieval. Recent methods in this area rely primarily on visual and textual features and often ignore audio, although it helps enhance overall comprehension of video content.Moreover, traditional models that incorporate audio blindly utilize the audio input regardless of whether it is useful or not, resulting in suboptimal video representation. To address these limitations, we propose a novel video-text retrieval framework, Audio-guided VIdeo representation learning with GATEd attention (AVIGATE), that effectively leverages audio cues through a gated attention mechanism that selectively filters out uninformative audio signals.In addition, we propose an adaptive margin-based contrastive loss to deal with the inherently unclear positive-negative relationship between video and text, which facilitates learning better video-text alignment.Our extensive experiments demonstrate that AVIGATE achieves state-of-the-art performance on all the public benchmarks.
Improving Text-based Person Search via Part-level Cross-modal Correspondence

Jicheol Park, Boseung Jeong, Dongwon Kim, and Suha Kwak

arXiv preprint, 2024

Abs arXiv

Text-based person search is the task of finding person images that are the most relevant to the natural language text description given as query. The main challenge of this task is a large gap between the target images and text queries, which makes it difficult to establish correspondence and distinguish subtle differences across people. To address this challenge, we introduce an efficient encoder-decoder model that extracts coarse-to-fine embedding vectors which are semantically aligned across the two modalities without supervision for the alignment. There is another challenge of learning to capture fine-grained information with only person IDs as supervision, where similar body parts of different individuals are considered different due to the lack of part-level supervision. To tackle this, we propose a novel ranking loss, dubbed commonality-based margin ranking loss, which quantifies the degree of commonality of each body part and reflects it during the learning of fine-grained body part details. As a consequence, it enables our method to achieve the best records on three public benchmarks.
PLOT: Text-Based Person Search with Part Slot Attention for Corresponding Part Discovery

Jicheol Park, Dongwon Kim, Boseung Jeong, and Suha Kwak

European Conference on Computer Vision (ECCV), 2024

Abs arXiv

Text-based person search, employing free-form text queries to identify individuals within a vast image collection, presents a unique challenge in aligning visual and textual representations, particularly at the human part level. Existing methods often struggle with part feature extraction and alignment due to the lack of direct part-level supervision and reliance on heuristic features. We propose a novel framework that leverages a part discovery module based on slot attention to autonomously identify and align distinctive parts across modalities, enhancing interpretability and retrieval accuracy without explicit part-level correspondence supervision. Additionally, text-based dynamic part attention adjusts the importance of each part, further improving retrieval outcomes. Our method is evaluated on three public benchmarks, significantly outperforming existing methods.
Efficient and Versatile Robust Fine-Tuning of Zero-Shot Models

Sungyeon Kim, Boseung Jeong, Donghyun Kim, and Suha Kwak

European Conference on Computer Vision (ECCV), 2024

Qualcomm Innovation Fellowship Winner
Abs arXiv Code Website

Large-scale image-text pre-trained models enable zero-shot classification and provide consistent accuracy across various data distributions. Nonetheless, optimizing these models in downstream tasks typically requires fine-tuning, which reduces generalization to out-of-distribution (OOD) data and demands extensive computational resources. We introduce Robust Adapter (R-Adapter), a novel method for fine-tuning zero-shot models to downstream tasks while simultaneously addressing both these issues. Our method integrates lightweight modules into the pre-trained model and employs novel self-ensemble techniques to boost OOD robustness and reduce storage expenses substantially. Furthermore, we propose MPM-NCE loss designed for fine-tuning on vision-language downstream tasks. It ensures precise alignment of multiple image-text pairs and discriminative feature learning. By extending the benchmark for robust fine-tuning beyond classification to include diverse tasks such as cross-modal retrieval and open vocabulary segmentation, we demonstrate the broad applicability of R-Adapter. Our extensive experiments demonstrate that R-Adapter achieves state-of-the-art performance across a diverse set of tasks, tuning only 13% of the parameters of the CLIP encoders.
HIER: Metric Learning Beyond Class Labels via Hierarchical Regularization

Sungyeon Kim, Boseung Jeong, and Suha Kwak

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

Abs arXiv Code Website

Supervision for metric learning has long been given in the form of equivalence between human-labeled classes. Although this type of supervision has been a basis of metric learning for decades, we argue that it hinders further advances in the field. In this regard, we propose a new regularization method, dubbed HIER, to discover the latent semantic hierarchy of training data, and to deploy the hierarchy to provide richer and more fine-grained supervision than inter-class separability induced by common metric learning losses. HIER achieves this goal with no annotation for the semantic hierarchy but by learning hierarchical proxies in hyperbolic spaces. The hierarchical proxies are learnable parameters, and each of them is trained to serve as an ancestor of a group of data or other proxies to approximate the semantic hierarchy among them. HIER deals with the proxies along with data in hyperbolic space since the geometric properties of the space are well-suited to represent their hierarchical structure. The efficacy of HIER is evaluated on four standard benchmarks, where it consistently improved the performance of conventional methods when integrated with them, and consequently achieved the best records, surpassing even the existing hyperbolic metric learning technique, in almost all settings.
Human Pose Estimation in Extremely Low-Light Conditions

Sohyun Lee^*, Jaesung Rim^*, Boseung Jeong, Geonu Kim, Byungju Woo, Haechan Lee, Sunghyun Cho, and Suha Kwak (*equal contribution)

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

Abs arXiv Code Website

We study human pose estimation in extremely low-light images. This task is challenging due to the difficulty of collecting real low-light images with accurate labels, and severely corrupted inputs that degrade prediction quality significantly. To address the first issue, we develop a dedicated camera system and build a new dataset of real low-light images with accurate pose labels. Thanks to our camera system, each low-light image in our dataset is coupled with an aligned well-lit image, which enables accurate pose labeling and is used as privileged information during training. We also propose a new model and a new training strategy that fully exploit the privileged information to learn representation insensitive to lighting conditions. Our method demonstrates outstanding performance on real extremely low-light images, and extensive analyses validate that both of our model and dataset contribute to the success.
ASMR: Learning Attribute-Based Person Search with Adaptive Semantic Margin Regularizer

Boseung Jeong^*, Jicheol Park^*, and Suha Kwak (*equal contribution)

IEEE/CVF International Conference on Computer Vision (ICCV), 2021

Bronze Prize at IPIU paper Award, Excellent Paper Award at BK21 Paper Award
Abs arXiv Code Website

Attribute-based person search is the task of finding person images that are best matched with a set of text attributes given as query. The main challenge of this task is the large modality gap between attributes and images. To reduce the gap, we present a new loss for learning cross-modal embeddings in the context of attribute-based person search. We regard a set of attributes as a category of people sharing the same traits. In a joint embedding space of the two modalities, our loss pulls images close to their person categories for modality alignment. More importantly, it pushes apart a pair of person categories by a margin determined adaptively by their semantic distance, where the distance metric is learned end-to-end so that the loss considers importance of each attribute when relating person categories. Our loss guided by the adaptive semantic margin leads to more discriminative and semantically well-arranged distributions of person images. As a consequence, it enables a simple embedding model to achieve state-of-the-art records on public benchmarks without bells and whistles.
Learning Discriminative Part Features through Attentions for Effective and Scalable Person Search

Jicheol Park^*, Boseung Jeong^*, Jongju Shin, Juyoung Lee, and Suha Kwak (*equal contribution)

IEEE International Conference on Image Processing (ICIP), 2020

Abs DOI

This paper proposes a new method for person search, the task of detecting a specific person exemplified by a query image from a gallery of scene images. Current state-of-the-art techniques in person search demonstrate impressive performance, but are limited in terms of efficiency and scalability since they require multiple models and/or have to re-process gallery images per query. We argue that a concise framework with a single neural network can achieve both of scalability and performance at once. In our framework, the network detects people and extracts their appearance features so that person search is done by finding the person closest to the query in the feature space. For performance, we focus on the quality of the person appearance features: Our network is designed and trained to produce person features that are discriminative, fine-grained, adaptive to appearance variations, and robust against person localization errors. To this end, we design channel attention and part-wise spatial attention modules as well as a loss for learning discriminative features. Our framework outperforms current state of the art on the PRW benchmark even with the concise pipeline based on a single network.