Let the deaf understand: Mainstreaming the marginalized in context with personalized digital media services and social needs

This paper presents a pilot study for a personalized media service which aims at creating an intelligent, sentiment-aware, and language-independent access to large archives of audiovisual documents, providing equal services to both mainstream and marginalized users. The proposed multi-modal framework analyzes aural, visual, and human descriptions, integrating them into an automatic content analyzer. Firstly, text is extracted from the aural stream and mapped to American Sign Language (ASL), translating conventional video to content suitable for the deaf. Next, sentiment is estimated from text, aural, and visual contents using two deep convolutional neural networks (CNN), extracting discriminative features from each modality. This provides output predictions for two broad classes: positive and negative sentiments. Preliminary results indicate that the proposed approach is capable of accurately estimating the sentiment of multimedia contents, which is an important step for personalized and intelligent media services.