We propose a simple but effective online ensemble tracking algorithm based on convolutional neural networks (CNNs). The proposed algorithm employs a CNN for target representation, which has a common convolutional layers but has multiple branches of fully connected layers. For better regularization, a subset of branches in the CNN are selected randomly for online learning whenever target appearance models need to be updated. We call this technique BranchOut. In addition, each branch may have a different number of layers conceptually to maintain variable abstraction levels of target appearances. BranchOut with multilevel target representation allows us to learn robust target appearance models with great diversity and makes it possible to handle various challenges related to target appearance modeling effectively. Our algorithm is tested standard tracking benchmarks and shows the state-of-the-art performance even without pretraining on tracking sequences.