Is a formal "science" of science policy an attainable (or desirable) goal?
Wed, 27/05/2009 - 15:00
As relative newcomer to the field of "science policy", I admit that I'm puzzled by one aspect underlying much of what I read. As someone trained in the natural sciences, I'm dogged by a nagging suspicion that many of the methods of the social sciences - in particular, the reliance on case studies and “expert judgment” - are... well... not as "rigorous" as what I'm used to. (My bias likely reflects the divergent metaphysical positions that are infused into natural and social scientists during their university training. For a fascinating discussion of this issue, see Marc Saner's recent paper "A Map of the Interface Between Science & Policy" at http://www.scienceadvice.ca/documents/Marc_Saner_Map_of_the_Interface_Between_Science_and_Policy.pdf ).
So, open question: can "science policy" be made more "scientific"?
In his 2008 report entitled "Evidence and Healthy Public Policy: Insights from Health and Political Sciences" (http://www.cprn.org/documents/50036_EN.pdf), Patrick Fafard writes (p15-16):
"The study of public policy was, for many years, thought by some to be an area of human endeavour that was a good candidate for careful scientific study. In the latter half of the 20th century, a movement of “policy science” emerged, which sought to develop a rigorously empirical approach to the study of public policy and testable theories of policy-making. [...] However, in the last 25 years or so, the discipline of policy studies has expanded considerably and moved well beyond the search for covering laws and testable theories of policy-making. Political and other social scientists who study public policy now embrace radically different theoretical and methodological approaches. [...] There are relatively few core or shared ideas among proponents of critical policy studies beyond a deep scepticism about the possibility or even the desirability of a science of policy-making..."
That said, it appears as if our colleagues in the U.S. are moving science policy in exactly that direction. In a 2008 report (http://www.ostp.gov/galleries/NSTC%20Reports/39924_PDF%20Proof.pdf), the U.S. Office of Science and Technology Policy moved to formalize and develop a "science of science policy" (p5):
“The science of science policy is an emerging interdisciplinary research area that seeks to develop theoretical and empirical models of the scientific enterprise. This scientific basis can be used to help government, and society in general, make better R&D management decisions by establishing a scientifically rigorous, quantitative basis from which policy makers and researchers may assess the impacts of the Nation’s scientific and engineering enterprise, improve their understanding of its dynamics, and assess the likely outcomes. Examples of research in the science of science policy include models to understand the production of science, qualitative and quantitative methods to estimate the impact of science, and processes for choosing from alternative science portfolios.”
The report identifies the pressing need for such an approach by noting that (p1):
"Although the importance of public investments in science, technology, and innovation is understood, the rationale for specific scientific investment decisions lacks a strong theoretical and empirical basis. Accordingly, given the magnitude of the Federal investment and the importance of that investment to our Nation, science policy decision makers must have at their disposal the most rigorous tools, methods and data that will enable them to develop sound and cost-effective investment strategies."
One of the main findings is that (p10):
"... while there is a well-developed understanding of human and social behavior in multiple disciplines such as economics, psychology, and sociology, this understanding has yet to be applied to the study of innovation within the scientific enterprise, leaving enormous gaps in scientific knowledge. For example: how does the discovery process work at the individual and team level? How could creative insights be stimulated? Which institutional structures facilitate the discovery to innovation cycle?"
In this view, a "science" of science policy may be an attainable goal... but we still have a long way to go.
Sun, 25/10/2009 - 15:00
#2
The limitations of a Science of Science Policy
Brian Wixted has stated many of the points that need to be made in this discussion. Falsification in the natural sciences is so successful because the systems studied are typically quite simple, they are under very high control, the investigations are very self-limiting, and the ethical issues virtually nil. This is why falsification is so successful in physics, much less so in biology (I am thinking of field studies here, for example), and even less so in psychology and medicine. Even in physics there are subfields -- cosmology and chaotic phenomena -- in which falsification is much more limited in its ability to rule out theories.
This is not to say that falsification is without value -- it should be used where it can be used. But it is to say that a healthy academic discipline will recognize methodological limitations, wherever such methods come from. Practitioners of sociology and political science should not expect the approaches of physics and chemistry to be transferable to their disciplines -- and neither should the natural scientists.
When it comes to setting policy -- which, as noted above, is not constant in time -- one must make the best use one can of the best results and approaches from all relevant disciplines.
Thu, 24/09/2009 - 15:00
#3
Refutability in science and social science
As a follow-up to my comment below:
I also wanted to add that this conclusion fits into Popper's general philosophy that the "scientific" is predicated upon its being refutable. Science, in Popper's view, proceeds through the process of trial and error, whereby some theories are discarded in favour of others that are simpler and have greater explanatory power. More importantly, though, these new theories should entail new predictions that can, at least in principle, be falsified. The principle of falsification is the vehicle of scientific progress, because it yields the only absolute certainty in science. We cannot ever prove a theory to be true, although we may find strong corroboration for it. We can, however, disprove a theory with a single observation. In discarding an incorrect theory, we make up a new and better theory to explain the new observations. Thus progress is made. If we forumulate any theory in a way that makes it non-refutable, the process of science ends. Popper says that such a theory is not scientific and, historically, is quickly superceded by a scientific theory.
It is often difficult to see how theories in the social sciences can be falsified. The scope, time frame, and complexity of social processes does not produce clear conditions that can practically be tested before the event they are meant to avoid or produce actually happens. Moreover, social sciences have a singular agenda to create actionable theories that are "true", because so much public opinion and finances depend on their success. Any sign of refutability would be a gap, so theories (and policies that derive from them) are necessarily worded in ways that make them non-refutable.
Falsification in the public policy domain is infamously seen as a failure, and it is difficult to see how progress can be made from a string of failures of policy to achieve promised results. As long as social science theories are conceived as endless forward-looking certainties in the absence of methods to corroborate or falsify them, they cannot lead to intellectual progress and, therefore, they remain unscientific (there may be some progress and regress in, say, the economy as we apply various policies, but how we think and form those theories remains unscientific).
Popper's proposal is that, although we cannot make robust scientific predictions in social sciences, we can make robust scientific conclusions about what we cannot do, by learning in a scientific way from the refutation of our past theories and policies.
Wed, 23/09/2009 - 15:00
#4
The status and purpse of the social sciences
Karl Popper's essays on the scientific status of the social sciences are quite illuminating and, I believe, highly applicable. He was a strong critic of what he called "historicism" and studied ways of applying rationalism and rigour to activities that are not strictly scientific (such as "traditions" and "social sciences"), often brilliantly analyzing both their scientific and un-scientific aspects.
In "Prediction and prophesy in the social sciences", he argues that the "doctrine which teaches that it is the task of the social sciences to predict historical development", is untenable -- largely due to the nature of society.
Popper sees the modern emphasis on prediction in general arising out humanity's early success in predicting astrological and meteorological events (seasons, floods). Such phenomena were indeed recurring and the methods simple. Unlike society. Otherwise, he said, “[h]ad a social scientist in 1780 known half as much about society as the old Babylonian astrologers knew about astronomy, then he should have been able to predict the French revolution".
In fact, “the most striking aspects of historical development are non-repetitive... Conditions are changing, and situations arise which are very different from anything that ever happened before.” He notes that the outcomes of human policies are often very different than the plans.
Popper makes the distinction between conditional "scientific prediction" and the "unconditional historical prophesies" of theoretical social sciences:
"Ordinary predictions in science are conditional. They assert that certain changes (say, of the temperature of water in a kettle) will be accompanied by other changes (say, the boiling of the water). Or to take a simple example from a social science: Just as we can learn from a physicist that under certain physical conditions a boiler will explode, so we can learn from the economist that under certain social conditions, such as shortage of commodities, controlled prices, and, say, the absence of an effective punitive system, a black market will develop. Unconditional scientific predictions can sometimes be derived from these conditional scientific predictions, together with historical statements which assert that the conditions in question are fulfilled... If a physician has diagnosed scarlet fever then he may, with the help of the conditional predictions of his science make the unconditional prediction that his patient will develop a rash of a certain kind. But it is possible, of course, to make such unconditional prophecies without any such justification in a theoretical science, or—in other words—in scientific conditional predictions. They may be based, for example, on a dream—and, by some accident they may even come true."
My contentions are two.
The first is that the historicist does not, as a matter of fact, derive his historical prophecies from conditional scientific predictions... he cannot possibly do so because long term prophecies can be derived from scientific conditional predictions only if they apply to systems which can be described as well isolated, stationary, and recurrent. These systems are very rare in nature; and modern society is surely not one of them.
He also points out the unempirical nature of the entities under study:
"[Take] the theory that the social sciences study the behaviour of social wholes, such as groups, nations, classes, societies, civilizations, etc. These social wholes are conceived as the empirical objects which the social sciences study in the same way in which biology studies animals or plants. This view must be rejected as naive. It completely overlooks the fact that these so-called social wholes are very largely postulates of popular social theories rather than empirical objects objects; and that while there are, admittedly, such empirical objects; and that while there are, admittedly, such empirical objects as the crowd of people here assembled, it is quite untrue that names like “the middle-class” stand for any such empirical groups. What they stand for is a kind of ideal object whose existence depends upon theoretical assumptions.”
So what is the nature of social sciences? And how can they be useful? Popper conclusion is that
“the main task of the theoretical social sciences [is] to trace the unintended social repercussions of intentional human actions. I may give a simple example. If a man wishes urgently to buy a house in a certain district, we can safely assume that he does not wish to raise the market price of houses in that district. But the very fact that he appears on the market as a buyer will tend to raise market prices. And analogous remarks hold for the seller.”
"The view that it is the task of the theoretical sciences to discover the unintended consequences of our actions brings these sciences very close to the experimental natural sciences. The analogy cannot here be developed in detail, but it may be remarked that both lead us to the formulation of practical technological rules stating what we cannot do. The second law of thermodynamics can be expressed as the technological warning, “You cannot build a machine which is 100 per cent efficient.” A similar rule of the social sciences would be, “you cannot, without increasing .productivity, raise the real income of the working population.” An example of a promising hypothesis in this field which is by no means generally accepted—or, in other words, a problem that is still open—is the following: “You cannot have a full employment policy without inflation.” These examples may show the way in which the social sciences are practically important. They do not allow us to make historical prophecies, but they may give us an idea of what can, and what cannot, be done in the political field."
Popper's article is from the book "Conjectures and Refutations", and is available at: http://keidahl.terranhost.com/Fall/HIS3104/Popper%20Prediction%20and%20Prophecy.pdf.
Wed, 29/07/2009 - 15:00
#5
A science of science policy
The notion of a ‘science’ of science policy is unfortunately constructed upon a number of assumptions and misunderstands the problem.
The first but not necessarily the most important is that of data. Science is constructed upon the idea of obtaining reasonably objective and repeatable results from repeated experiments. Those who propose a science of science policy unfortunately assume that we can get to close to the same in the social world. This is simply not the case. Take for instance innovation surveys – essential for developing such a science. The answers you get from such surveys have been shown to be dependent upon who fills them in. The Australian ABS actually went back for an internal project and scrutinised a large random sample to reveal than the R&D department, finance department and CEO have quite differing opinions within the same organisation. Even if we could reliably get the raw data of organisations we are faced with how to organise the results. Organisational forms are very different from standardised industry classifications. Ordering the data into categories is necessary but not simple for developing the simplified view necessary for the science proposed.
The second challenge is that such an approach assumes a degree of stability and transferability of policies. This ignores the uniqueness of place and very largely the innovation systems literature of the last twenty years. Canada is not the USA, the USA is not Europe etc. Further the USA or Canada of today are in very different global positions to that of even of twenty or thirty years ago. Geography, industry structure, cultural history and time are all important in the evolution of science policy for a particular place.
Lastly, the science of science policy perspective is not actually constructed on the ideals of the natural sciences but on the idea of having models of similar ‘power’ to those in micro-economics (see Marburger 2006 http://www.oecd.org/dataoecd/48/14/37483994.pdf). And personally I don’t find that an attractive proposition.
All this is not to say that we can’t improve science policy we can. But policy is of the moment - it is the deals that trade off genuine conflicting pressures in employment, hospital funding and industry at a particular point in time given a particular set of constraints.
Mon, 14/09/2009 - 15:00
#6
Science of Science Policy
While I agree with both the previous posts, i would like to add a note from one corner of the science policy world, that I, as a former molecular biologist (does one ever really recover?), finds the closest to the kind of data gathering/ generating studies I was familiar with in the lab.
The corner of the science policy world about which I am speaking is grounded in anthropological research methods, so, instead of relying on what people say or write about what they do, which is notoriously unreliable, focusses on analysis of artefacts (i.e. graphs, gels, diagrams, etc), video evidence, and in situ "field reports" - so, in an ideal study, you have a number of different information sources that triangulate to support or disprove various points.
As you can imagine, the conclusions that derived from the studies of scientists at work as they created and re-created models, hypotheses, and in some cases, the objects of study (i.e. new tissue/cell cultures) - look quite different from the ones generated by surveys of CEOs. I found these studies to be more useful in understanding how science can, does, and sometimes does not make a difference in the world.
Unfortunately, the earliest proponents of these methods got caught up in the silly "science wars" of the mid-90s - some may say they only have their philosophically-minded writing style to blame - but since then, there have been more and more research that loses the references to specialized language and insights from metaphysics and cultural studies, and communicates the ideas in more straight-forward ways. Authors such as Brian Wynne, Jeff Bowker and Michael Lynch come to mind. You may want to check these out to get a slightly more concrete handle.
Of course, the master commentator on science and power, Bruno Latour, may try the patience of a natural science practitioner, although the "Pasteurization of France" and "Aramis" (especially given these times' fascination with new infrastructure) may prove interesting reading.Organizing Committee
Mehrdad Hariri (chair)
Shiva Amiri
Marcius Extavour
Eleanor Fast
Curtis Forbes
Minnie Kim
Jeffrey Kinder
Odile Lagacé
Trevor McKee
Robin McLernon
Anton Neschadim
Jeff Sharom
Mahadeo Sukhai
Ilia Tikhomirov
Masoud Yeganegi
Christine Zhang
Shiva Amiri
Marcius Extavour
Eleanor Fast
Curtis Forbes
Minnie Kim
Jeffrey Kinder
Odile Lagacé
Trevor McKee
Robin McLernon
Anton Neschadim
Jeff Sharom
Mahadeo Sukhai
Ilia Tikhomirov
Masoud Yeganegi
Christine Zhang
Twitter (sciencepolicy)
Discussion of the "science of science policy" seems to omit (up to now) that the Canadian government did indeed adopt the new discipline of science policy in 1964 (Science Secretariat in the Privy Council Office) or 1971 (Ministry of State for Science and Technology) and later folded these agencies into others (and abolished the more effective Science Council besides) -- suggesting that, at least a generation ago, the experiment was attempted and abandoned as a failure.
Discussions of future policy might be strengthened (and abbreviated) by awareness of how often our predecessors found themselves in just the situation we know today and misapprehend as new, such as anxiety about the relative strengths of the university, governmental and industrial R&D communities (discussed Oct. 26 on TV Ontario by Conference Chairman Hariri.) This was the top policy question in 1945 (as documented by public speeches of NRC President C.J. Mackenzie) no less than at later dates (e.g. of the Glassco Inquiry 1960-61, the Lamontagne Report of the early 1970s, the Science Policy Forum, and so on.)
Industrial R&D deserved top priority in the postwar years, Mackenzie recommended; this did not happen by itself, so (in the decontrolled postwar economy) the policy was changed, to develop instead governmental and (a little later) university research. The Glassco and Lamontagne Reports evoked stronger-phrased policies, by governmental ministers rather than scientific leaders, that industrial R&D demanded top priority and would get maximal support from governmental decisions, e.g. the "Make or Buy" policy to contract out government's own research needs, e.g. IRDIA. If after nearer 40 than 30 years the results are still unsatisfactory we may profitably consider just what features of Canadian research make it so unamenable to direction.
Secondly, we may learn from the actual dynamics of governmental decisions concerning science, known to be an arcane and complicated domain in which there are no votes. There is no evidence that actual Canadian politicians ever wanted "science policy advice." This is merely something theoreticians preach that politicians ought to want, i.e. "really, really need." By contrast, we could instead review the actual processes by which cabinet-level or interdepartmental decisions about science are really made. Theoreticians and "advisers" usually have up their sleeves something (like Leontief input-output matrices) that they find truly fascinating but that politicians find either too complicated to understand or too far from their interests to be practical. This appears to be a continuing fact of the Canadian political process.
Statistics demonstrate the case. In the 1970s Canadian science statistics became possibly the best in the world for science policy planning. But they were seldom if ever used for this purpose, and were expensive, so they were largely abandoned. If a future generation of science policy advisers demands statistics they will have to reinvent the wheel --after first securing a budget to do so. (This too is not new. The Advisory Panel on Science Policy made a special trial of science statistics in the 1950s, collecting data in two domains, medical and industrial, to see whether the federal policy regime of the day could make timely and profitable use of them. It could not, all participants agreed, so the experiment halted. This case escaped the attention of the Glassco Commissioners, and therefore of the (Lamontagne) Senate Committee because of its decision to review no history before 1962.)
Dr. Hariri pleaded cogently Oct. 25 for a "national conversation" about science, through which laymen could reach scientists and researchers could reach the science policy community. He seemed unaware just how often this has begun before now, through the magazine Science Forum which lasted a decade until it passed into governmental hands and collapsed, through CBC radio and television, the National Forum on Science Policy, etc. This history might be profitably studied so far as it suggests a repeating pattern: that strenuous work generates such a national conversation every 10 or 15 years, but all have failed either to continue alive (cf. Science Forum) or to satisfy either the scientists who take part or the public figures who might be deemed the main beneficiary of group wisdom concerning science policy.
Dr. P.D. McTaggart-Cowan said more than once that he saw secret lists of government priorities twice during his years as Secretary of the Science Council of Canada. Science appeared on one of the lists of priorities, in last place, and in the other list was not mentioned at all. This seems to be the normal status of science in political life. The main decision facing Canadian science policy analysts may be whether they can afford to ignore this normal ambient condition.