A key issue in system identification is how to cope with high system complexity. In this contribution we stress the importance of taking the application into account in order to cope with this issue. We define the concept cost of complexity which is a measure of the minimum required experimental effort (e.g. used input energy) as a function of the system complexity, the noise properties, and the amount, and desired quality, of the system information to be extracted from the data. This measure gives the user a handle on the trade-offs that must be considered when performing identification with a fixed experimental budget. Our analysis is based on the observation that the identification objective is to guarantee that the estimated model ends up within a pre-specified ``level set” of the application objective. This geometric notion leads to a number of useful insights: Experiments should reveal system properties important for the application but may also conceal irrelevant properties. The latter, dual, objective can be explored to simplify model structure selection and model error assessment issues. We also discuss practical issues related to computation and implementation of optimal experiment designs. Finally, we illustrate some fundamental limitations that arise in identification of structured systems. This topic has bearings on identification in networked and decentralized systems.