Attend and Guide (AG-Net): A Keypoints-driven Attention-based Deep Network for Image Recognition


Research output: Contribution to journalArticle (journal)peer-review

31 Citations (Scopus)
541 Downloads (Pure)


This paper presents a novel keypoints-based attention mechanism for visual recognition in still images. Deep Convolutional Neural Networks (CNNs) for recognizing images with distinctive classes have shown great success, but their performance in discriminating fine-grained changes is not at the same level. We address this by proposing an end-to-end CNN model, which learns meaningful features linking fine-grained changes using our novel attention mechanism. It captures the spatial structures in images by identifying semantic regions (SRs) and their spatial distributions, and is proved to be the key to modelling subtle changes in images. We automatically identify these SRs by grouping the detected keypoints in a given image. The ``usefulness'' of these SRs for image recognition is measured using our innovative attentional mechanism focusing on parts of the image that are most relevant to a given task. This framework applies to traditional and fine-grained image recognition tasks and does not require manually annotated regions (e.g. bounding-box of body parts, objects, etc.) for learning and prediction. Moreover, the proposed keypoints-driven attention mechanism can be easily integrated into the existing CNN models. The framework is evaluated on six diverse benchmark datasets. The model outperforms the state-of-the-art approaches by a considerable margin using Distracted Driver V1 (Acc: 3.39%), Distracted Driver V2 (Acc: 6.58%), Stanford-40 Actions (mAP: 2.15%), People Playing Musical Instruments (mAP: 16.05%), Food-101 (Acc: 6.30%) and Caltech-256 (Acc: 2.59%) datasets.
Original languageEnglish
Pages (from-to)3691-3704
Number of pages14
JournalIEEE Transactions on Image Processing
Early online date11 Mar 2021
Publication statusPublished - 11 Mar 2021


  • Computer Vision
  • Deep Learning
  • Attention Mechanism
  • Self-Attention
  • Semantic Regions
  • Scale Invariant Feature Transform (SIFT)
  • Fine-grained visual recognition
  • Artificial intelligence
  • Convolutional Neural Network

Research Centres

  • Centre for Intelligent Visual Computing Research
  • Data Science STEM Research Centre
  • Data and Complex Systems Research Centre


Dive into the research topics of 'Attend and Guide (AG-Net): A Keypoints-driven Attention-based Deep Network for Image Recognition'. Together they form a unique fingerprint.

Cite this