The problem of current English assessment in Taiwanese classroom settings was visited. A recommendation to assess English ability in the notion of semantic and pragmatic competence was made. An instrument was constructed in line with our recommendation and administered to 341 middle school and college students. We hypothesized that in terms of item percent correct, items measuring both semantic and pragmatic competence will become easier as students progress from middle school to college; whereas in terms of item discrimination, semantic items are more discriminating than pragmatic items in the current context. Empirical analyses confirmed our hypothesis and the implication discussed.