Procedures 

As part of their agreement with FCPS, Sun Associates provided the district with all the materials and procedures necessary to replicate its original assessment.  As much as possible, those same procedures were followed in the 00/01 and 01/02 DTAs.  In some cases, the procedures were further refined in order to (a) make adjustments that Sun Associates recommended or (b) improve the reliability and validity of last year’s procedures.  The majority of this year’s data were derived from three main sources: 

1. An online survey
2. Classroom observations
3. Focus groups

Certain strengths and weaknesses are associated with each of the three methods of gathering data.  For example, even though classroom observations allow assessors to record specific teaching and learning behaviors in a real-life setting, they are plagued by the fact that they only give the assessor a snapshot of what actually occurs in the classroom throughout the school year.  That weakness is balanced out by the fact that the online survey and the focus groups both give the assessor a broader, yearlong view of teaching and learning behaviors.  The process of balancing three types of data collection procedures to compensate for certain weaknesses is known as “triangulation.” 

Online Survey 

An online survey of teachers was conducted in November and December.  All teachers were asked to complete the 10-minute survey in a computer lab during a regularly scheduled staff meeting.  Individual survey responses by teachers were kept confidential, so teachers would feel able to express themselves freely.  However, at the request of SBDM councils, their school’s cumulative survey results may be reported to the council as a means for suggesting school-wide improvements in technology implementation. 

At the suggestion of Sun Associates, two items were added to the 00/01 survey that were not included in the 99/00 survey.  They were “12. Software for testing student achievement (e.g., STAR Math or SRI)” and “54. The administrator in my school is supportive of technology.”  Also, four items were changed in order to clarify the questions.  In the 99/00 survey, it was possible for a teacher to inadvertently select a response they did not intend to select.  That was corrected on the 00/01 survey.  The complete list of survey items can be seen in table 2 in the results section, and the changed items are indicated in that table.

Classroom Observations 

Classroom observations were done between November and January to provide more direct data on technology progress.  Technology Resource Teachers (TRT) conducted the observations, concentrating on classroom activities that relate to the rubrics developed for the baseline DTA.  The observations were done in 105 classrooms in 57 elementary, 28 middle, and 20 high schools.

In any educational research, it is important to take steps to ensure “reliability” of data collected in classroom observations.  That is, it is important to safeguard against the possibility of the observer missing important information or recording information inaccurately.  Several steps were taken to counteract any potential “human error” problems.  Each TRT was paired with another TRT, and they compared notes after their observations to be sure they agreed on what teaching and learning behaviors they saw.  Everyone used the same observation form (see appendix 3) and the forms were designed to facilitate collecting as much objective data as possible.  Every attempt was made to avoid subjectivity.  The TRTs were assigned to observe in schools where they don’t normally work, as a way to avoid any potential biases.  TRTs received training on how to use the observation forms, and then conducted a trial run, to make sure the forms were used correctly. 

In each participating school, the School Technology Coordinator (STC) was asked to identify two classrooms where they feel technology is being used most effectively.  Two additional classrooms were identified randomly.  One additional observation was done in the school’s computer lab.  That mix of settings was intended to show the best the school has to offer in technology, balanced with typical use of technology.  It also gave both the classroom and lab views.  However, the emphasis was definitely on classroom observations rather than computer labs because that gives a more realistic view of a student’s typical daily schedule.  The observations lasted 30 to 60 minutes each.

Focus Groups 

Focus groups were held to further clarify the survey and observation data.  The focus group questions are in appendix 2.  The groups consisted of: 

• Technology Resource Teachers
• Students
• Elementary
• Middle
• High

Timeline

Table 1.  Data Collection Timeline