🎨 Mask-OVEN-Wiki Dataset for PL-VEL

Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking (AAAI 2025)

[GitHub] [Paper] [Hugging Face]

📖 Overview

Mask-OVEN-Wiki is an automatically annotated dataset designed for pixel-level visual entity linking, extending the OVEN dataset. It introduces pixel-level masks for entities in images to enable fine-grained visual grounding. Mask-OVEN-Wiki dataset includes 5,245,421 annotations across 5,214,965 images from the OVEN dataset, covering 20,077 distinct entities. We retained OVEN's knowledge base, which contains 6,063,945 Wikipedia entities, with 2,032,340 entities having corresponding images. This setup bridges text-image alignments at both entity and pixel levels.

📥 Download Data

1. Mask-OVEN-Wiki Annotations

Available on Hugging Face: NP-NET/mask-oven-wiki

Dataset	Split	Size	Images	Mentions	Entities	Link
Entity Set	Train	12.5 GB	4.46M	4.46M	7.9k	📥 Download
Entity Set	Val	498 MB	108k	108k	3.2k	📥 Download
Entity Set	Test	2.97 GB	608k	608k	15.9k	📥 Download
Query Set	Train	96.2 MB	19.8k	23.5k	2.5k	📥 Download
Query Set	Val	8.3 MB	1.8k	1.9k	632	📥 Download
Query Set	Test	58.1 MB	14.1k	15.4k	5.4k	📥 Download
Human Set	-	25.5 MB	3.6k	4.3k	2.3k	📥 Download
Wiki Set	-	5.37 GB	2.03M	1.97M	2.03M	📥 Download

2. OVEN Images & Knowledge Base

• Images: Follow OVEN guidelines.
• Knowledge Base:
  • Full Info (6.9G)
  • Title Only (419M)

📌 Example

{
    "images": [{"id": "oven_04925844", "file_name": "/04/oven_04925844.jpg", "height": 333, "width": 500}],
  "categories": [{"id": "Q388450", "name": "Crow"}],
  "annotations": [
      {
          "id": "oven_query_val_00000107",
      "image_id": "oven_04925844",
      "category_id": "Q388450",
      "segmentation": [{"size": [333, 500], "counts": "..."}, ...],
      "bbox": [[92.76, 11.17, 498.73, 331.54], ...],
      "text_prompt": "A organisms known by a particular common name named 'crow'...",
      "label": ["a organisms a bird(0.44)", "the genus corvus(0.31)"],
      "question": "what kind of bird is this?",
      "data_split": "query_val_seen"
    }
  ]
}

🚀 Usage

Step 1: Install Dependencies

pip install pycocotools

Step 2: Load Data with COCO API

import json
from pycocotools.coco import COCO

# Load annotations
ann_file = "path/to/mask_oven_query_val.json"
coco = COCO(ann_file)

# Explore images
image_ids = coco.getImgIds()[:3]
for img_id in image_ids:
    img = coco.loadImgs(img_id)[0]
    print(f"ID: {img['id']}, File: {img['file_name']}, Size: {img['width']}x{img['height']}")

# Explore annotations
ann_ids = coco.getAnnIds(imgIds=image_ids)
anns = coco.loadAnns(ann_ids)
for ann in anns:
    print(f"Label: {ann['label']}, Bbox: {ann['bbox']}, Mask: {ann['segmentation']}")

Step 3: Decode Masks

from pycocotools import mask as mask_util

# Decode RLE to numpy array
rle = anns[0]['segmentation'][0]
mask = mask_util.decode(rle)

# Merge multiple masks
def merge_masks(rles):
    masks = [mask_util.decode(rle) for rle in rles]
    return np.bitwise_or.reduce(masks)

mask_merged = merge_masks(anns[0]['segmentation'])

📝 Citation

If you use Mask-OVEN-Wiki, please cite:

@article{xu2024reverse,
  title={Reverse Region-to-Entity Annotation for Pixel-Level Visual Entity Linking},
  author={Xu, Zhengfei and Zhao, Sijia and Hao, Yanchao and Liu, Xiaolong and Li, Lill and Yin, Yuyang and Li, Bo and Chen, Xi and Xin, Xin},
  journal={arXiv preprint arXiv:2412.13614},
  year={2024}
}

🌟 Acknowledgments

Special thanks to the OVEN dataset, GroundedSAM, SEEM for foundational work!
For issues, contact @NP-NET-research or @xuzf-git.