How will I know when I am successful?
(Faculty discussion at AIRE Assessment Workshop, Jul 1999)

Students show intrinsic interest in the discipline and/or research and engagement with the work/discipline.
Individual faculty members said they knew they were successful when:
Students ask more questions in class.
There is an improved classroom dynamic.
Students are involved and committed to the material covered in class.
Students show enthusiasm for the research process.
The students take a deeper interest in the subject matter (greater enthusiasm, want to pursue it more).
The student's sense of connectedness to other and to the discipline increases.
The student's interest in the field increases; there is a feeling of excitement and wonder.
Students tell me they will pursue independent research partially because of the course.
Students go on to take another math or science course that they were not planning on taking.
Students show more interest in the life of the intellect.
Students appreciate how many disciplines are important (physics, comp science, chemistry, math, engineering overlap).
When students are fully engaged in the work.
When students want to learn and do more with their topic.
Hypothesis: An increase/change in interest will lead to greater persistence in the discipline, choice to do research, choice of discipline-based career. Students will want to learn more and will enjoy the learning.

Students make connections, integrate knowledge/ideas and create new ideas from these. They show depth in their analysis of problems and understanding of complexity.
Individual faculty members said they knew they were successful when:
Students actually achieve/reach original conclusions/ideas in their research.
Students become comfortable integrating information from the literature.
Students learn to integrate various concepts and information.
New thoughts/ideas emerge as the result of discussion and as a result of research activities.
Students say "Ah Ha!"
Students demonstrate the ability to analyze problems and issues at a deeper and more complex level.

Students can argue from data, defend their conclusions, critically evaluate the research of others, see logical errors, see flaws/strengths in own/others work.
Individual faculty members said they knew they were successful when:
Students can defend their conclusions.
When students pinpoint errors/inadequacies in my research design/hypothesis.
When students can critically evaluate someone else's research (i.e. find flaws and good aspects).
When students can detect logical fallacies.

Students show they can "think like a scientist". They question and show skepticism.
Individual faculty members said they knew they were successful when:
Students understand the implications of inherent constraints in the world that involve inevitable trade-offs.
Students doubt/question their preconceptions and assumptions.
Students begin thinking like scientists. In particular, they ask "What is the evidence for that assertion, etc". They show skepticism.
Students can understand and identify on an exam the nature of science as a way of knowing versus other ways of knowing. They appreciate the benefits of answering a question using science. They can differentiate good science from pseudo science or non-science.

They are comfortable with and appreciate uncertainty, complexity and questioning.
Individual faculty members said they knew they were successful when:
Students become comfortable with asking questions related to unknowns (hypothesis generation).
Students appreciate the complexities that go into designing a new material.

Students show understanding of the scientific process. They ask questions, can generate hypotheses, design an experiment, evaluate data and redesign an experiment.
Individual faculty members said they knew they were successful when:
Students can turn an idea into a research topic and design.
Students can design an experiment with proper controls.
When most students identify the many reasons why their planned experiments did not work/did not answer their experimental hypothesis.
When most students develop their own reasonable/testable research hypothesis.

Students handle data like a scientist. They show competence in entering data, reading data, analyzing data, graphing data, collecting data. They can use the computer for all of these. They recognize a reasonable measurement. They can ballpark/estimate a measurement.
Individual faculty members said they knew they were successful when:
Students can enter data and calculate inferential statistics using SPSS.
Calculate descriptive statistics using Excel.
Students can translate a graph into words.
Students can generate follow up research ideas from a presentation of the literature.
Students can prepare a simple graph using Excel.
Students can enter data on Excel.
Students can read off a graph and make reasonable conclusions about what it says for the material.
Students can make a measurement with a reasonable feel for uncertainty and accuracy of the measurement.
Students can correctly answer questions on an exam regarding the basic logic of inferential statistics.
Students can correctly compute the mean, median, and standard deviation of a data set.
Students can examine the frequency distribution of a data set and correctly determine whether the sample was drawn from a normal distribution.

Students have good presentation skills; they show confidence and professionalism.
Individual faculty members said they knew they were successful when:
Students become comfortable with data presentation.
When students can clearly present experimental findings.
The student's self confidence in their scientific ability increases.

Students write in a professional style/quality.
Individual faculty members said they knew they were successful when:
Students can write an APA-style results section.
Students can write papers approaching professional caliber.
Students can write in scientific style.

Students report that they are well prepared for the next class or stage in their career.
Individual faculty members said they knew they were successful when:
A student returns after her first year of graduate school and says she had no trouble with the theory.

Students engage with complex data and find patterns (independently). They think creatively about data, observations, literature sources.
Individual faculty members said they knew they were successful when:
Students think creatively and independently.
Students muck around in primary sources and find direction without direct professorial intervention-cope with "open-endedness" of historical research and analysis.
Students read the literature to improve their hypothesis or their experimental techniques.

Students show understanding of lab/computer techniques well enough to trouble shoot.
Individual faculty members said they knew they were successful when:
Students can troubleshoot problems.
Students understand the technique they are using well enough so they can figure out what to do if something goes wrong.

Students can turn a qualitative statement of a problem or observation into a quantitative statement/model. Also the reverse; be able to turn quantitative results into statements of findings, arguments.
Individual faculty members said they knew they were successful when:
Students can turn "English" questions into quantitative analyses and perform the appropriate tests.
Students can successfully model a problem on their own or in groups, demonstrated via a report, etc.
Students are aware that mathematical modeling is an iterative process.
Students can take a real world problem, construct a mathematical model of it, use software to solve the mathematical model, and use the solution to say something useful about the original problem.

Students carry forward into the future. They may go on to do an independent project and work collegially with the professor, get into graduate programs, and/or get research jobs/careers.
Individual faculty members said they knew they were successful when:
The student and I give a joint talk or publish a paper together.
Students successfully apply what they have learned elsewhere; on the job, in grad school.
Students go on to do independent research or honors work.