A small exploratory study was conducted over a school term in three primary schools in the Northern Territory to determine the effectiveness of a web-based literacy intervention, ABRACADBRA. Our purpose is to inform others of the challenges encountered while implementing and evaluating the effectiveness this new literacy intervention. Researchers were confronted with several ongoing obstacles during the data collection process. Using standardised testing with children who had never been exposed to this type of assessment; working with teachers with very little teaching experience and poor attendance all added to the difficulty of conducting rigorous research. Tests that were designed for whole class administration had to be given one-on-one or to small groups of no more than three. Teacher turnover was an issue as was the fact that most teachers were early in their careers, still developing knowledge and skills. While many of the teachers were eager to learn and responsive to professional development, their overall lack of training in explicit early literacy instruction at times negatively impacted the program's delivery. Greater teacher support is suggested to increase overall teacher effectiveness while providing much needed professional development. Sporadic attendance among many of the participants resulted in the study losing 22 of its original 97 students over a ten week period. Computer assisted learning programmes such as ABRACADABRA, which allows students to pick up where they left off is one way of addressing the issue of transient students. ABRACADABRA, with its multiple levels and variety of activities allows teachers to differentiate lessons according to student needs. Recommendations for other educational researchers working within this or a similar context are given throughout the article.