Data Analysis
Project improvement plan
Calls for practices commonly referred to as “data-driven decision-making (D3M)†have grown in recent years. In practice, D3M privileges quantitative data such as transcripts and course-taking, attendance, achievement, and participation in a range of programs. In theory, D3M practices allow leaders to digest a large volume of information to inform a range of important decisions such as selecting curricula; starting, continuing, and/or ending interventions; and how to allocate scarce resources. And while the kinds of quantitative data that are well-suited to D3M practices have become more widely available, data, and analyses of data, do not “speak for themselves.†Thus, it is imperative that leaders be able to thoughtfully engage with data to inform their practice and to engage with others’ arguments that appeal to data. The purpose of this project is to provide students with experience in practical data analysis for organizational (school, district, network, system, state, etc.) improvement.
Through the components of this project, you will draft an organizational improvement plan in which you identify a problem of practice in/for a specific organization (school, district, network, system, state, etc.), identify and use a quantitative dataset to describe and characterize it, conceptualize an inquiry to learn more about the factors that produce the problem in that organization, draw on extant research to propose an intervention to address the problem, and craft a presentation on your work. This project will be submitted in parts so that feedback can be provided by your instructor(s) along the way to inform subsequent steps.
The components of this project are outlined below:
Part I – Quantitative Analysis
This part of the project consists of several subsections: a discussion of the problem of practice, a description of a quantitative dataset that can be used to explore this problem, and a quantitative analysis that draws on this dataset to identify and discuss the important elements and/or dimensions of this problem. Importantly, this should be done in prose (not bullet points) and follow APA formatting.
Problem of Practice (2-3 pages)
A problem of practice is an issue wherein one or more organizational outcomes fall short of the organization’s goals. Depending on one’s position, an organization may be a department, school, district or charter network, regional service provider, or even a state education agency. A problem of practice can be approached or defined in several ways, and the way in which it is approached shapes how the rest of the work is framed. For instance, a leader may compare an organization’s progress against its strategic plan or goals to identify areas where improvement is needed. One taking this approach would begin by identifying the organization’s goals (or at least a relevant subset of them) and measuring progress toward them, keeping an eye toward whether any particular patterns exist in regards to successes and failures. A leader may also identify a problem of practice by considering how larger social phenomena (e.g. discrimination, equity, declining postsecondary attendance) may be playing out within their organization. One taking this approach would begin by reviewing larger issues that may effect their organization; thinking through which issues are most important to, or most likely to affect, them; and then using available data to evaluate the extent to which the identified issues affect their organization.
The description of your problem of practice should do the following:
- Clearly identify the organization that is the focus of your improvement plan project.
- Discuss the topic(s) that will be the focus of your improvement plan. It is unlikely that you will be able to assemble a comprehensive improvement plan that addresses all underperforming areas, so you will have to make, and defend, decisions about the areas on which to focus.
- Motivate this work by discussing the relevant strategic goals or larger social issues that are relevant to your problem. This should include citations to specific documents or pieces of evidence. That is, it should refer to a document that contains the organization’s goals or literature that outlines the general social problem along with a discussion of why those may pertain to your organization.
- Be written in prose. Supplemental text features such as graphics, tables, or bullet points may be used if necessary, but the problem of practice should be crafted as a narrative.
Data Description
To investigate your problem of practice, you will need to identify a relevant administrative dataset that can be used to unpack or investigate it. An administrative dataset is a set of quantitative data that are collected and assembled as part of an organization’s regular operations. Administrative data may include variables such as measures of student achievement, student absences, number of students enrolled, teachers’ qualifications and years of experience, school funding, or a range of other measures. Administrative [quantitative] data are required here because quantitative data and methods allow one to generalize to a population, which permits one to speak to the scale and scope of an issue and are thus crucial to understanding issues at the organizational level. The description of your data should include the following:
- Who collected the data and how
- What is the population represented by the dataset (e.g. all 3rd graders at Harriet Tubman Elementary or all public school districts in NJ, NY, and PA) the unit of observation is (is it student-level, teacher-level, school-level, district-level, state-level, etc.)
- What time period is represented by the data
- What information is captured by the data (you do not need to list all of the variables in the dataset, but should discuss the kinds of variables that it contains)
- How many observations are in the dataset
- A discussion of how this dataset will be useful for investigating your problem of practice, as well as its potential limitations (one almost never has “perfect dataâ€)
You are free to select the dataset that is most applicable to exploring your problem of practice. One dataset will be provided as will information on how to find others.
Quantitative Analysis
After describing your dataset, you will present a quantitative analysis of factors/variables that are relevant to understanding your problem of practice, using visuals or graphics as appropriate. As a first step in this, you will likely need to operationalize your problem of practice in a way that lends itself to quantitative analysis. This involves identifying specific organizational outcomes from your problem of practice, describing them, and statistically evaluating relationships that are relevant to the problem of practice. A good first step should be to show relevant descriptive statistics for the organization you are studying and for the overall sample in your dataset. This helps to show the extent to which your organization is “typical†or more of an outlier in different areas. From there, your quantitative analysis may proceed in different ways, such as using descriptive statistics, Z-scores, simple hypothesis tests like t-tests, ANOVA, or correlations, or by using regression. For instance, say that one’s problem of practice involves postsecondary enrollment among students from a specific high school. Part of the quantitative analysis could include one or more of:
- A comparison of the high school’s postsecondary enrollment rate against a goal in the school improvement plan
- A comparison of the high school’s postsecondary enrollment against other high schools statewide using percentiles or Z-scores
- Constructing a confidence interval around the school’s postsecondary enrollment rate to evaluate whether it is similar to the statewide average
- Correlations between the postsecondary enrollment rate and: the % of the student body identified as economically disadvantaged, the % of students who took the SAT, and the % of teachers with at least a master’s degree to identify factors that are related to this outcome
- Multiple regression that simultaneously considers the relationships between multiple factors and postsecondary enrollment
- Using the residuals from a multiple regression to examine how well the high school performs after considering its observable characteristics
When developing your quantitative analysis, consider the variables that you are using as these determine which statistical approaches are appropriate. Also, note that datasets at different “levels†often have similar variables coded differently. For example, student-level data will have race/ethnicity and gender coded as categorical variables while datasets at the school-level or above will have these variables coded as continuous variables since the variable will be a percentage or proportion of students in that category.
- Rubric elements
- Problem of practice is well-described (3)
- Problem of practice is situated in larger social context of education or specific improvement goals (3)
- Suitability of the data to the problem is addressed
- Data is accurately and adequately described (2)
- Data Analyzed are relevant to the problem of practice (1)
- Appropriate analyses are used (2)
- Quantitive results are adequately and accurately reported (2)
- Appropriate use of tables and figures (2)
Part II – Intervention Proposal
In this part of your improvement plan you will propose at least one intervention to address your problem of practice. This intervention needs to be informed by both your understanding of your problem of practice and by relevant literature. The first component of Part II is a discussion of your working theory or hypothesis of your problem of practice based on your quantitative analysis and what you think your survey is likely to show. In other words, describe what you believe is going on that shapes your problem of practice. Here, some conjecture is appropriate so long as it is logically justified (think “if…, then†statements), but you should not make claims that are not warranted by what you include in Part I. For instance, if one is examining achievement gaps, it is reasonable to conclude something along the lines of “if the curriculum/instruction that is used to educate students is divorced from their experiences and perspectives, they may be less successful and so a lack of culturally relevant curriculum and teaching may contribute to this achievement gap†is logical, while something along the lines of “these achievement gaps are driven by differences in socioeconomic status†is less logical without additional evidence such as measures of SES and a consideration of whether SES varies more within or across schools/districts.
After outlining your working theory or hypothesis, discuss at least 2-3 pieces of research that are helpful in understanding your problem of practice. At least one of these sources should be an evaluation that tests whether an intervention was successful in improving some outcome. When selecting sources, you should seek out those that are peer-reviewed if they are in a journal, that are from a reputable publisher if a book, or if it is a report, that it is authored by reputable researchers or is published by a reputable institution. Sources from periodicals, blogs, vendors, and advocacy organizations should not be used because they are more prone to bias and less rigorously evaluated. Your discussion of these sources should include their data and method, main conclusions, and how they inform what actions may address your problem of practice.
Lastly, Part II should propose an intervention to address, at least in part, your problem of practice (“silver bullets†are rare, so it is unlikely that a single intervention will in short order completely alleviate a problem). This proposed intervention could be similar to one evaluated in earlier research (e.g. that you review above). Alternatively, if you identify an evaluation that was not successful (and there are many), this is still informative because it suggests that one should consider interventions of a different nature. Importantly, along with being informed by the research you discuss, this proposed intervention should be related to your working theory of your problem of practice, which itself is rooted in your earlier analyses.
- Rubric elements
- Working theory is logical (3)
- Working theory is thoroughly described (3)
- At least one piece of research referenced is a quantitive evaluation (1)
- Pieces of research are appropriately and accurately discussed (4)
- Proposed intervention is logical and connected to the theory of the problem and quantitive analyses
Part III – Presentation
Prepare a recorded, engaging, presentation on your improvement plan that you will deliver to your classmates virtually after our last set of class meetings. Your presentation should walk the audience through the problem of practice you are investigating and the data you have selected, describe the variables of interest, review your quantitative analysis and present/interpret results clearly and concisely, review your working theory of your problem of practice, describe your proposed intervention, and discuss how a survey could be used to further explore your problem of practice to confirm or challenge your working theory of it.
Specifically, using Parts I and II, along with feedback from your instructor, create a slide deck for a presentation, of 12-15 minutes in length, on your improvement plan that is designed to be delivered to an audience of stakeholders of your choice. For example, you may choose to create a presentation that could be given to a school board, administrative team, or colleagues. While the presentation should draw on earlier parts of this project, it should not be built by simply copying and pasting. It may be helpful to think of your presentation as an opportunity to tell the story of your work: why your problem is important, how the data you selected can help understand it, why your methods are well suited to exploring the problem, what your administrative data show or imply about your problem of practice and what remains unknown, your working theory of your problem, how your proposed intervention(s) will address the problem. Lastly, Part IV requires you to develop a survey to further explore your problem of practice. Your presentation must include a section that outlines your thinking on how a survey could complement your quantitative analysis by generating evidence that supports or challenges your working theory of your problem. In discussing your proposed survey, describe the relevant constructs that you would need to measure to learn more about your problem of practice, how knowledge of those constructs would complement your quantitative analysis, and potentially show several example items that you could include on your survey.
At the outset of your presentation, briefly describe your intended audience to help your classmates and instructors understand the context for this presentation and provide more meaningful feedback. In the body of your presentation, text should be used sparingly. Avoid reading from slides. Visuals are generally preferred when possible and should clearly illustrate the relationships you are investigating. The number of slides in each presentation may vary, but there should be enough to keep the audience’s attention but not so many that the audience struggles to keep up. A good rule of thumb for presentations is 1-2 minutes for each slide that contains content.
To make your virtual presentation, you will need to make a recording of you giving your presentation using Teams, PowerPoint, etc. The recording will then be posted to the course Teams page where your classmates will be able to view it and provide comment/feedback. An important part of this component of the project is to gain experience both giving and receiving critical feedback, so comments should be substantive, focusing on how the data are used, analytical choices, potential alternative theories of the problem, and interpretation.
*(Please just create the powerpoint and I will present)
- Rubric elements
- Presentation is clear and coherent (2)
- suitability of the data to the RQ’s is addressed (2)
- Variables used in the analysis are adequately described (2)
- Rationale and constructs for the survey is clearly articulated (2)
- Results are thoughtfully presented and interpreted using visuals appropriately (2)
- Constructive comments are provided for at least 3 other presentations (3)
Part IV – Survey
Quantitative data is very useful to understanding the prevalence and degree or organizational issues. However, administrative datasets rarely contain the type of information that can be used to understand the mechanisms and perceptions that shape, or even produce, these issues. For this, additional inquiry via surveys or qualitative work can be helpful. For this part of your improvement plan, you will develop a survey to explore an element or facet of your problem of practice that cannot be addressed through your quantitative analysis. The aim of this survey will be to shed light on some element of your problem of practice in a way that will generate potential solutions. For this, you do not have to collect data, you only have to assemble an instrument and a plan for data collection.
The first step in developing this will be to discuss, in prose, the ways in which your quantitative analysis informed your view of your problem of practice – what you learned through that and what remains unknown. What remains unknown will then guide the development of your survey. Next, you will discuss the constructs that are relevant to furthering your understanding of your problem of practice and how they will do so. Third, you will develop a survey that can be completed in approximately 10 minutes that measures these constructs and include that as an attachment. Fourth, you will describe an administration plan that does the following:
- Identifies the target population
- Discusses how a sample will be drawn from the target population if the whole population cannot be surveyed
- Describes how the survey will be distributed to the sample
- Describes how data will be collected from respondents
Each element of the administration plan should be aligned with your problem of practice and the unknown issues you identify after your quantitative analysis. Additionally, it should be logical and practical. If for instance, one is using a survey to gauge the extent to which students in a given district have access to technology (internet and a laptop or desktop) in their home, a survey that is administered to a sample via phone is more sensible than a survey administered via email since those without access to technology would be unlikely to receive an invitation to participate in the survey. Finally, you will describe how this survey will be analyzed to gain insight into your problem of practice. Things to consider for this description are:
- At what level will the survey data be and what are the implications of that for understanding your problem of practice? For instance, your administrative dataset may be school-level while your survey is student-level, which allows for different kinds of inferences that should be considered.
- Will survey responses be linked with administrative data and used for more sophisticated analyses?
- What analyses will be used for this survey data (e.g. descriptive statistics, t-tests, correlations, ANOVA, regression, factor analysis, etc.)? In selecting analyses, be sure to consider whether variables are categorical or continuous as this informs the kinds of analyses that are appropriate.
- Rubric elements
- Analysis of what can be reasonably concluded, and not, from quantitative analysis (4)
- Constructs on the survey are clearly described(2)
- Survey is of the appropriate length (1)
- Survey uses appropriate scales (2)
- sampling and administration of survey are adequate and clearly described (3)
- Analysis plan for survey data is appropriate (3)
Part V – Final Improvement Plan
The final part of this project is an APA formatted document, written in prose, that integrates Parts I, II, and IV, with revisions based on instructor feedback and peer feedback from the presentation and that includes a discussion of how your working theory could change in response to survey data if you were able to field/administer it. That discussion should address the following:
- What kinds of responses would give you more confidence in your working theory?
- What kinds of responses would weaken your confidence in your working theory?
- What may lead you to abandon your working theory in search of a new one?
The final report should be formatted using headings and subsections to break the document up into more manageable “chunks†and so that readers can more readily identify sections of interest. The final report should also be included tables and figures that support the text (using APA format for these as well). Lastly, an APA formatted references section should also be included.
- Rubric elements
- Report uses proper formatting (2)
- Instructor feedback is incorporated (3)
- Feedback from the presentation is incorporated (2)
- Parts 1-3 is coherently incorporated into a single piece of writing (4)
- Discussion speaks back to RQ’s addresses limitations and thoughtfully identifies next steps (4)