Evans School of Public Policy & Governance

The AP Science Impact Study, which is funded by the National Science Foundation, seeks to understand the impact of Advanced Placement Biology and Chemistry classes on the high school students who take them. It examines the effects of the updated inquiry-based curriculum on students’ confidence in scientific inquiry skills and their post-high school plans, including college type, selectivity, and major. This study has policy implications for science curriculum and the next generation STEM workforce.

Study Motivation 

AP Science & Inquiry-Based Learning

Advanced Placement courses are designed to pre­pare high school stu­dents for the rigor of col­lege course­work. The curriculum was revised by The College Board in 2012 and 2013 to bet­ter develop stu­dents’ abil­ity to con­duct sci­en­tific inquiry, emphasizing critical thinking over factual recall. The current AP classes are designed to give students confidence in their scientific inquiry skills, hopefully leading them on to further scientific study.

STEM & Policy Implications

Strengthening our teaching and learning of science, engineering, and quantitative reasoning is important for the workforce readiness of students, the growth of the U.S. economy, and our ability to develop innovative solutions to increasingly complex policy problems. This study will help guide policymakers and funders to implement effective, large-scale educational policy and funding decisions and help guide educators in advising individual students and prioritizing particular classes and curriculums.

Study Design

Experimental Design

While non-experimental studies have shown large, positive effects of advanced high school courses, we seek to fill in the gaps by using an experimental design to control for other variables. Most educational studies are observational in nature and do not involve a control and treatment group that can be compared with one another. With a randomized experimental design, we can control for extraneous variables and make greater inferences about causation instead of just correlation.

Hypothesis

The cen­tral hypoth­e­sis of our pro­posed research is that taking an inquiry-based AP® Biol­ogy or Chem­istry course will cause stu­dents to be more inter­ested and com­pe­tent in sci­en­tific inquiry, more likely to enroll in and com­plete col­lege, and more likely to pur­sue and per­sist in STEM majors. We fur­ther expect that stu­dents with less prior prepa­ra­tion will have more to gain from the inquiry-based AP® course and, thus, receive greater aca­d­e­mic ben­e­fits than students with more prior prepa­ra­tion. Finally, we expect larger effects in class­rooms where the cur­ricu­lum is imple­mented with a high degree of fidelity.

Project Components

The AP Science Impact study involves several different components to analyze our hypothesis, including the newly created scientific inquiry instrument, student surveys, teacher and administrator surveys and in-person interviews, classroom observations, and administrative student data. The project is funded by the National Science Foundation, partners with researchers at George Washington University, Equal Measure, and other institutions, and is done in collaboration with The College Board.

Papers and Takeaways for Practice

Conger, D, Long, MC, McGhee Jr., R. “Advanced Place­ment and Initial College Enrollment: Evidence from an Experiment.” Forthcoming in Education Finance and Policy

To evaluate how Advanced Placement courses affect college-going, we randomly assigned the offer of enrollment into an AP science course to over 1,800 students in 23 schools that had not previously offered the course.

  1. We find no AP course effects on students’ college entrance exam scores (SAT/ACT).
  2. As expected, AP course-takers are substantially more likely to take the AP exam than their control group counterparts. At the same time, treatment group students opt out of the exam at very high rates and most do not earn a passing score on the AP exam.
  3. Though less precisely estimated, the results also suggest that taking the AP course increases students’ aspirations to attend higher-quality colleges but does not lead to enrollment in such institutions.

Conger, D, Kennedy, A, Long, MC, McGhee Jr., R. “Effects of Advanced Placement Science Courses on Skill, Confidence, and Interest in Science.” The Journal of Human Resources, 56, 93-124, 2021.

Compared to the control group, AP course-takers, on average:

  1. Scored slightly higher on a science assessment and expressed more interest in pursuing a STEM major in college: Overall, the authors find some evidence that AP classes increases students’ skill in science. Indeed, both students and teachers report that AP courses are more challenging that non-AP science courses. On an end-of-year assessment designed by the researcher team to measure scientific inquiry skills, AP science students scored slightly higher. However, the difference between the two groups was not statistically significant. The assessment measured students’ skill in data analysis, scientific explanation and scientific argument, rather than specific knowledge in Biology or Chemistry. Students in both the AP course and control group were very likely to say they were interested in a STEM degree, with the AP course takers being somewhat more likely on average, though the difference between the two groups was not statistically significant. Sixty-two percent of the control group and 71 percent of AP course takers said they are interested in a STEM degree should they enroll in college.
  2. Expressed lower confidence in their ability to succeed in college science: Overall, those wishing to take the AP science course had very high confidence in their ability to succeed in a college science course. AP course-takers however, were slightly less likely than those in the control group to be at least somewhat confident in their ability succeed in a college science course (82 percent vs 92 percent of the control group).
  3. Experienced higher levels of stress: Taking AP science has a strong negative effects on stress levels of students; 29 percent of AP students reported that their most recent science class had negative impacts on their stress levels compared to 12 percent of those in the control group.
  4. Had worse grades: On average, students who take an AP science course earn slightly lower grades in science (-0.3 grade points) and slightly lower grades in their other courses (-0.18 grade points). While institutions tend to upweight AP grades due to their relative academic difficulty, the authors find that the weighting should be greater than what is typically done. They find that applying a weight of 1.5 (instead of 0.5 or 1, as many institutions do) would be needed to fully adjust for the effect of taking an AP science course on a students overall GPA.

Long, MC, Conger, D, McGhee Jr., R. “Life on the Frontier of AP Expansion: Can Schools in Less-Resourced Communities Successfully Implement AP Science Courses?” Educational Researcher 48(6), 356 –368, 2019.

This paper evaluates the implementation of AP Biology and Chemistry courses across 23 schools in 11 school districts across the nation. The study revealed several impediments in the implementation process, of which teachers and administrators wishing to pursue the introduction of such courses may find useful:

  1. While teachers tended to welcome the addition of the AP classes and express excitement to teach the courses, many indicated having difficulty finding the time to prepare class content and desired more support from the school for developing the curriculum. The factors most associated with difficulty preparing class included not having a master’s degree, not actively engaging the class in discussion and debate, and dissimilarity between instruction method and AP science approach.
  2. Despite all students having met the pre-requisites for the offered courses, teachers reported a large variation in student preparedness at the onset of the class, hindering the speed at which the content could be taught. Teachers also reported that students were unprepared for the self-taught “inquiry-based” learning of the courses. To overcome these obstacles, some teachers reporting using scaffolding techniques to allow students to gradually shed the need for outside assistance. Others assigned content typically covered in class as homework to allow for enough time to cover all of the course material.

Seeratan, K, McElhaney, KW, Mislevy, J, McGhee Jr., R, Conger, D, Long, MC. “Measuring Students’ Ability to Engage in Scientific Inquiry: A New Instrument to Assess Data Analysis, Explanation, and Argumentation.” Educational Assessment, 25(2), 112-135, 2020.

The science education community lacks assessments that reliably and validly measure knowledge, skills, and abilities that are needed to support engagement in authentic science inquiry. In this paper, we describe the conceptualization, design, development, and testing of an instrument focused on measuring students’ proficiency at the intersection of data analysis, scientific explanation, and scientific argument.

Researchers and practitioners may choose to use the validated instrument as is, if it fits their particular need or may choose to use it as a starting point for developing similar instruments relevant to their own research contexts. The strengths of the instrument lie in its relatively brief time for administration and the minimal amount of science disciplinary knowledge required for students to answer the questions. Our work has value in modeling how to design assessments that align to the National Research Council Framework, adhere to the recommendations of the measurement community, and forefront the assessment of science practices while still integrating them with disciplinary knowledge following contemporary science education frameworks. Our work also informs the design of Next Generation Science Standards-aligned assessments.

Related Links

Research Team

Principal Investigators

Mark C. Long is a Professor of Public Policy and Governance and an Adjunct Professor of Economics at the University of Washington. He is affiliated with several centers on campus including Scholars Strategy Network, Center for Education Data & Research, West Coast Poverty Center, and the Center for Studies in Demography and Ecology.

Dr. Long’s research examines the effects of public policies (particularly education policy) on economic opportunity and efficient social mobility, with emphasis on estimating the benefits and costs of those policies. He was the recipient of the American Educational Research Association’s Palmer O. Johnson award for “the highest quality of academic scholarship published in one of the [four] AERA journals during the 2008 volume year” and co-recipient of the 2014 Wilder School Award for Scholarship in Social Equity and Public Policy Analysis for advancing the public’s understanding of social equity. He is a co-editor of Journal of Policy Analysis and Management and vice president of the Policy Council of the Association for Public Policy Analysis and Management.

Dr. Long holds a Ph.D. and M.A. in Economics from the University of Michigan. He also holds an M.P.P. from the University of Michigan, and a B.A. from DePauw University.

Dylan Conger is a Professor and Director of the Masters in Public Policy program at the Trachtenberg School of Public Policy and Public Administration. She is also a research affiliate at the George Washington Institute of Public Policy and New York University’s Institute for Education and Social Policy.

Dr. Conger’s research concerns disadvantaged, immigrant, and minority youth with a focus on education policies and urban areas. She is currently a managing editor of the Journal of Policy Analysis and Management, has published in numerous journals, and is on two advisory committees for the U.S. Department of Education. She has served on the Board of Directors of the Association for Education Finance and Policy and the Editorial Boards of Educational Evaluation and Policy Analysis and Educational Research. She is also a Technical Panel Member for the National Center for Education Statistics’ National Assessment of Educational Progress High School Transcript Study. Before joining the faculty of the Trachtenberg School, Dr. Conger held research positions at the Vera Institute of Justice and Abt Associates, Inc.

Dr. Conger received her Ph.D. in Public Policy from New York University, her M.P.P. from the University of Michigan, and her B.A. in Ethnic Studies from the University of California at Berkeley. View bio: Dylan Conger

Raymond McGhee Jr. is a Senior Program Officer at the Robert Wood Johnson Foundation where “he works across various RWJF program areas to advise on key evaluation efforts, including a long-term evaluation of the Foundation’s systems change efforts in building a Culture of Health.”  Previously, Dr. McGhee served as Senior Director at Equal Measure, where he directed and contributed to the evaluations of several national multi-site initiatives in the fields of college access and success, youth economic development, family and economic security, and STEM.  Prior to Equal Measure, he was a Senior Research Scientist in SRI International’s education division, conducting research and program evaluations on efforts to improve students’ transition from secondary school to postsecondary school and to the workforce. McGhee has worked with states, districts, community-based organizations, and schools to plan and implement formative and summative evaluations. He has also been involved in studying recruitment, retention, and induction programs designed to prepare postsecondary students for careers in STEM.

McGhee received his Ph.D. in Education, M.A. in Applied Linguistics, and B.A. in Linguistics from the University of California, Los Angeles.

Research Staff

Dr. Alec Kennedy was a Research Assistant and earned a Ph.D. at the Evans School of Public Policy and Governance at the University of Washington. He has a general interest in education policy research and the economics of education. His work focuses on understanding the effectiveness and implications of education policies with a particular focus on the impacts of such policies on economically disadvantaged and at-risk student populations. He has earned a M.S. in Public Policy and Management from the University of Washington and B.A.s in Economics and Statistics from the University of California, Berkeley.

Kelsey Rote was a Research Analyst at the Evans School of Public Policy and Governance at the University of Washington. She is interested in education policy and research spanning all ages, from early learning to adult education. Prior to joining the Evans School, she taught English as a Peace Corps Volunteer in Ukraine, analyzed risk and protective factors for high school students in Central and South Seattle, and created foreign language curricula using infant and child brain development research. She earned an M.S.W. in Administration and Public Policy from the University of Washington and B.A.s in Psychology and Sociology from Seattle Pacific University.

Nicole Bateman was a Research Assistant and Master of Public Administration student in the Evans School of Public Policy and Governance at the University of Washington. Her policy interests are diverse and include climate and energy, urban, and education policy. Prior to attending the Evans School, Nicole spent several years with a technology-consulting firm and also worked with the League of Conservation Voters to advocate for strong environmental policy and candidates for public office. She earned B.A.s in English and Political Science from Seattle University.

External Evaluator

Michal Kurlaender served as our evaluation consultant for this research study. Kurlaender is a Professor and Chair at UC Davis’ School of Education. She investigates students’ educational pathways, in particular K-12 and postsecondary alignment, and access to and success in postsecondary schooling. She has expertise on alternative pathways to college and college readiness at both community colleges and four-year colleges and universities. Kurlaender also studies the impact of racial and ethnic diversity on student outcomes, and is an expert on the dismantling of desegregation plans and persistent inequalities in schools. She serves on the executive committee of the UC Davis Center for Poverty Research, is the UC Davis site director of the Educational Evaluation Center, and is a researcher with the Center for Analysis of Postsecondary Education and Employment at Columbia University. Kurlaender received her Ed.D. and Ed.M. from Harvard University Graduate School of Education and B.A. from the University of California, Santa Cruz. View bio: Michal Kurlaender

Advisory Board

Richard Murnane, an economist, is the Thompson Research Professor at the Harvard Graduate School of Education and a research associate at the National Bureau of Economic Research. Murnane is well-known for his expertise in education policy as well as randomized controlled trials in the field of education. Alongside his colleague Greg Duncan, Murnane recently published Restoring Opportunity: The Crisis of Inequality and the Challenge for American Education (2014). View bio: Richard Murnane

Helen R. Quinn is Professor Emerita of Particle Physics and Astrophysics at Stanford’s SLAC National Accelerator Laboratory, and has taught physics at both Harvard and Stanford. Dr. Quinn is an internationally recognized theoretical physicist who holds the Dirac Medal, the Klein Medal, the Compton Medal, and the Sakurai Prize. She is a member of the American Academy of Arts and Sciences, the National Academy of Science, and the American Philosophical Society, and a Fellow and former president of the American Physical Society. She served as Chair of the US National Academy of Science Board on Science Education (BOSE) and has served as a member of the BOSE study that formed the basis of the Next Generation Science Standards (NGSS) that have been adopted by multiple states in the US. Dr. Quinn was recently appointed by the President of Ecuador as a member of the initial commission to help guide the development of the National University of Education of Ecuador.

Aaron Rogat has a Ph.D. in Molecular Cell Biology and is involved in science education curriculum development, assessment, and policy. He is a Research Associate at Purdue’s Department of Educational Studies and a former Research Scientist at Educational Testing Service (ETS). Rogat is interested in using advances from the learning sciences to develop tools that support the teaching and learning of science. Over the past several years his research and development efforts have focused on the development and use of learning progressions and innovative assessments for teachers and students, many of which are computer-based. He also has specific interests in supporting the learning of biology and the ability to develop and use scientific models. View bio: Aaron Rogat

Liaison with The College Board

Tanya Sharpe (Senior Director of AP Science) will serve as an information resource to the research team regarding the results of Board research on developments in the AP science curriculum and professional development activities for AP.

Partners

Partners

We have 23 schools in 11 districts across the country.  District partners include: Charlotte-Mecklenburg (NC), Winston-Salem/Forsyth (NC), Cranston Public Schools (RI), East Side Union (CA), Lynwood Unified (CA), Jefferson Parish (LA), Richmond (VA), Metro Nashville (TN), Anaheim Union (CA), El Paso ISD (TX), Detroit EAA (MI).

This research project is done in partnership with the researchers at the Evans School of Public Policy and Governance at the University of Washington, Trachtenberg School of Public Policy and Public Administration at George Washington University, Equal Measure, and SRI International.