Neural networks are often used for learning classifiers for problems, where a structured solution also exists. However, even in that case the explainability of the neural-network solution is typically very limited, compared to the structured solutions. The task of this thesis is to design and investigate approaches to explain the computational process in the neural network as follows. We assume we are given higher-level concepts, which are relevant according to some external knowledge such as domain knowledge or another, more structured solution. First, we search for these concepts as possibly occurring in the network as a combination of neurons and design the respective algorithms. Second, we evaluate (1) how precisely the concepts are represented in the neural network (also in comparison to irrelevant concepts), (2) how the concepts are used in the computation of the neural network to reach the final decision (in particular statistically). The thesis will provide the necessary theoretical investigation as well as a prototypical implementation evaluating how accurately the theoretical concepts describe the behaviour occurring in actual neural networks.