We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks (RNN) over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. ...
Tópico(s): Domain Adaptation and Few-Shot Learning
2016 - IEEE Computer Society | IEEE Transactions on Pattern Analysis and Machine Intelligence
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, Li Fei-Fei,
The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, ...
Tópico(s): Image Retrieval and Classification Techniques
2015 - Springer Science+Business Media | International Journal of Computer Vision
Richard Socher, Andrej Karpathy, Quoc V. Le, Christopher D. Manning, Andrew Y. Ng,
Previous work on Recursive Neural Networks (RNNs) shows that these models can produce compositional feature vectors for accurately representing and classifying sentences or images. However, the sentence vectors of previous models cannot accurately represent visually grounded meaning. We introduce the DT-RNN model which uses dependency trees to embed sentences into a vector space in order to retrieve images that are described by those sentences. Unlike previous RNN-based models which use constituency ...
Tópico(s): Natural Language Processing Techniques
2014 - Association for Computational Linguistics | Transactions of the Association for Computational Linguistics
Andrej Karpathy, Michiel van de Panne,
Humans and animals acquire their wide repertoire of motor skills through an incremental learning process, during which progressively more complex skills are acquired and subsequently integrated with prior abilities. Inspired by this general idea, we develop an approach for learning motor skills based on a two-level curriculum. At the high level, the curriculum specifies an order in which different skills should be learned. At the low level, the curriculum defines a process for learning within a ...
Tópico(s): Human Pose and Action Recognition
2012 - Springer Science+Business Media | Lecture notes in computer science
Stelian Coros, Andrej Karpathy, Ben Jones, Lionel Revéret, Michiel van de Panne,
We develop an integrated set of gaits and skills for a physics-based simulation of a quadruped. The motion repertoire for our simulated dog includes walk, trot, pace, canter, transverse gallop, rotary gallop, leaps capable of jumping on-and-off platforms and over obstacles, sitting, lying down, standing up, and getting up from a fall. The controllers use a representation based on gait graphs, a dual leg frame model, a flexible spine model, and the extensive use of internal virtual forces applied ...
Tópico(s): Robotic Locomotion and Control
2011 - Association for Computing Machinery | ACM Transactions on Graphics
2017 - IEEE Computer Society | IEEE Transactions on Pattern Analysis and Machine Intelligence