Real World, Real stories: teaching quantitative methods with real life data more

REAL WORLD, REAL STORIES: TEACHING QUANTITATIVE METHODS WITH REAL LIFE DATA C. Russell1, S.Noble2 J. Carter3, S. Currier4, R.Wiseman5 University of Manchester (UNITED KINGDOM) University of Manchester (UNITED KINGDOM) 3 University of Manchester (UNITED KINGDOM) 4 Sarah Currier Consultancy Limited (UNITED KINGDOM) 5 University of Manchester (UNITED KINGDOM) E-mails celia.russell@manchester.ac.uk, susan.noble@manchester.ac.uk, jackie.carter@manchester.ac.uk, sarah.currier@gmail.com, richard.wiseman@manchester.ac.uk 2 1 Abstract Recent work has shown that using real-life, regularly updated data to teach econometrics and related social science statistical skills has a number of benefits over using fictional or pre-configured datasets that have been developed purely for specific learning activities. The use of actual data adds interest and relevance to courses, helps to prepare students for using statistics in the real world and allows them to explore recent events. Based on case studies collected through the “Real World: Real Data: Real Stories project”, ESDS International at Mimas has developed the multimedia, Teaching Tools elearning resource. This web-based resource is designed for social science lecturers who are using, or plan to use, real data for the first time in their teaching. It is intended to support and encourage lecturers' use of real world data in their courses - by presenting examples of how this is already done and hearing actual lecturers speak about their experience of using this type of 'messy' real-world data in their teaching. Mimas, a National Centre of Excellence based at the University of Manchester is home to ESDS International. Part of the wider Economic and Social Data Service, ESDS International provides the UK further and higher education communities with free access to and support for a huge range of socio-economic international data. The Teaching Tools e-learning resource is part of our wider engagement with the statistical literacy agenda and our work to build skills around the use of real world data in research-led teaching. In this paper we discuss the development of the Teaching Tools resource, describe the case studies on which it was based and report on the benefits and barriers of using real, live data in undergraduate and postgraduate teaching. We also highlight some of the challenges of using real world data in teaching that remain ahead. Keywords: Teaching, learning, quantitative methods, real data, economics teaching, econometrics, evidenced-based learning, pedagogies, constructionism 1 INTRODUCTION The move towards evidenced–based policy making has highlighted concern about levels of data and statistical literacy skills of UK social science students, and led to many calls to improve research-led teaching in the area [1]. Students often avoid data handling and analysis with the result that many social science students graduate with only a limited experience of managing quantitative data. This is paradoxical as years of strategic investment into data infrastructures have provided the UK academic community with access to a rich set of social science data resources. The challenge for educators lies in promoting students' use of data, but the benefits in doing so improve both academic performance and job prospects for students. Efforts in the UK have focused on developing learning activities in university teaching programmes to address this challenge. In 2009, ESDS International at Mimas carried out the Real World: Real Data: Real Stories project which used a case study approach to examine how educators have successfully used real life data in their undergraduate and postgraduate teaching to upskill students in data handling and its disciplinerelated usage[2]. 2 METHODOLOGY ESDS International disseminates and supports the large time series databanks produced by international governmental organisations such as the World Bank and IMF. For this project, semistructured interviews were carried out with economics lecturers from the Universities of Manchester and Loughborough who use data from ESDS International in their undergraduate or taught postgraduate courses. The interviews were conducted face-to face and asked lecturers (i) how they use the data in teaching (ii) to what purpose (iii) whether it enhances the skills base of their students (iv) whether they had evidence of feedback from their students (v) whether they could comment on the skills with regards to employability and (vi) what advice they could offer for improving the use of national data services in learning and teaching [3]. Econometrics is often considered a dry and dusty subject but the courses explored in the case studies were highly rated by students. In this way, the case studies provide details of successful attempts to make learning and teaching with data for econometrics at undergraduate and postgraduate level a less passive and more engaging experience. One interviewee even reported the use of words like "enjoyable" and even "fun" in feedback from students. He continued: “I do get very good feedback for this. Econometrics is not a popular subject because we have loads of students who find it technically hard. But I think the feedback I get is that they actually quite enjoy doing the project because it is actually something practical and they can see that there may just be an application for this that they'll use when they leave here.” (Paul Turner, Reader in Economics and Postgraduate Research Programme Director, Loughborough University in interview, 10 July 2009) An additional purpose of the research was to enhance understanding of how national data services, such as ESDS, support and build the skills base of the nation’s undergraduate and postgraduate community. Accordingly, one focus of the interviews was to gather qualitative evidence through narratives about how these real world data services are utilised within taught courses, and how they impact on the student experience. 3 3.1 THE CASE STUDIES Case study 1: Applied Economics 1st year undergraduate module at University of Manchester Three existing courses and one proposed course formed the focus of the research. ESDS International is used in a practical exercise within the Applied Economics module, which held in the second semester. As part of the specialist Bachelor of Economic Science degree, the aim of the module is to introduce students to how economic data can be used to analyse economic hypotheses and issues. In this module students undertake some applied economics research, in a very structured, hand-held manner, without really knowing much about econometrics. They are introduced to things that they will learn in-depth later on, with the aim of showing them that they can produce the kind of project they might be able to see being asked for in the workplace. This, along with familiarising them with econometrics software packages, and refreshing their understanding of tasks such as matrices and matrix inversion in Excel, helps spark their interest and build confidence early on. Students are given a lecture introducing the particular problem of estimating a consumption function for the UK, with an explanation of the relevant statistical techniques and economic theory. “The purpose of this course was to get them doing some applied economics research without them really knowing anything, and so what we did was specify a particular problem, estimate the consumption function for the UK, explaining the statistical techniques, explaining the economic theory, and statistical techniques that you've used, and I gave them all a chapter from a very nice book by Roger Backhouse, in which he does that and so it goes through a series of different equations, different models which become increasingly more elaborate and then he estimates each of those in turn and in a way explains the economic theory about why one was, each one was not so, what the problems were with it and so really all have to do is reproduce that chapter using up-todate data.” (Paul Turner, in interview, 10 July 2009) In IT lab tutorials, students are shown, by working step-by-step through the chapter, how to reproduce the multiple regression model and update it to the present. The data they need for updating the model is extracted live from ESDS International, and combined with data from the Office of National Statistics and the former Office of the Deputy Prime Minister, the last of which has house price data. Thus they are required to pull data from three different sources, combine it in a spreadsheet and update an existing consumption model. This exercise, while gently spoon-feeding the students to some extent, is useful for showing undergraduates how raw data may need to be manipulated. For instance, the house price data series from the Office of the Deputy Prime Minister does not start at the same period as the other data, but there is another series that can be tagged onto it to create a single time series, with some manipulation at the overlap of the two series to make them consistent. This also gives the students an opportunity to learn to footnote how this manipulation was done, which emphasises correct documentation methods, key for scientific replicability. 3.2 Case Study 2: Introduction to Econometrics, Second Year undergraduate course at Loughborough University The module Introduction to Econometrics is taught to second year economics undergraduates at Loughborough University. The aim of the module is to understand basic and more advanced econometrics techniques that will allow economic theories to be tested using econometrics applications and tools. With around 170 students, the course is fairly large. The project begins with an introductory lecture on ESDS International and how to use it, with guidelines also posted on the Loughborough virtual learning environment. The lecturer has also created some short animations, freely available on YouTube, to assist students with various tasks. Each student is given an individual topic, for example the relationship between consumption and income in one of seven different countries (Australia, Belgium, Denmark, France, Germany, Italy, Japan, UK or US). With 170 students, this means there is reasonable variety of topics, while ensuring the lecturer is able to support them if required. Students are then expected to download annual time series data from IMF's International Financial Statistics (via ESDS International), organize it in Excel, export into eViews, run a regression, estimate some diagnostic statistics and then write up a short report. Having so many students tackling projects on a individual basis does add to the teaching load of the course. Each student has their own particular problems and so the lecturer does not take students through this process in tutorials, nor does he give email support; he finds it easier to have extra open office hours during the period when students will be working on their assignments. “I put on extra office hours when the project's due in because almost always they want individual attention. And what I find is that when I'm doing this, I put on these extra office hours, I will have for that hour I'll have a queue of students right back down that corridor.” (Paul Turner, in interview, 10 July 2009) Students often run across some statistical problems with what they estimate and require some feedback to help them understand how to deal with this. It's part of the process of learning with realworld data that they comprehend the possible difficulties that arise even when all the steps are undertaken correctly. Students are required to submit for assessment a very tightly defined, fairly short report, which will be 20% of the total course mark. This will include a regression report and diagnostics, with a couple of pages interpreting the statistics and economics, with a table to check their actual work. The fact that this project has obvious applicability to possible tasks and skills required in the workplace is a clear incentive for students; in his interview the course lecturer observed: “I always start the econometrics course off by telling them that if there's one course they may actually make use of when they leave- it's probably going to be this one.” (Paul Turner, in interview, 10 July 2009) 3.3 Case study 3: An Applied Development Economics Project at the University of Manchester The University of Manchester offers an MSc in Development Economics and Policy. One of the course units is the Applied Development Economics Project (ADEP). The core aim of ADEP is to provide students with practical experience of applying real world data in a practical, empirically-driven project. There are eight lectures and five lab tutorials (each with a single tutor and around 20 students), and the unit is assessed by a single project dissertation at the end. Students attend introductory lectures which refresh their memory on how to do research in an applied piece of work, and how to find data (see Case Study 1 above for First Year introduction to this work). At this point ESDS International and other sources, such as the Office for National Statistics, are demonstrated, with an overview of what these sites do and how to extract data from them. Tutorials are then held in IT lab sessions. In the past there were five such tutorials given, although this number has been reduced recently. Students begin with a tutorial giving a recap of econometrics techniques; appropriate software packages that they might want to use are specified. Choosing the right econometrics package to use in labs is important: some are not robust enough for multiple simultaneous use, and crash all the machines in an IT lab; some are too expensive for students to purchase to play with at home. However, learning to learn to work with different software packages can be a bonus; once graduates are in the workplace they may well be confronted with unfamiliar tools that they need to learn quickly. In a second tutorial students work to specify their own research problem and halfway through the unit, a project proposal is submitted, although this is not assessed. There are always a few students who are overly ambitious, and the tutor sometimes needs to give them guidance on simple projects, e.g. by reproducing a textbook example. Occasionally students want to work with data from a country or on a topic that requires data sources that the tutor is unfamiliar with; in this case students need to try out their own searching techniques. In further tutorials, the students get data they need for their project, estimate some simple models to explore, begin to try explaining relationships, and look at how to report back on their project findings. Some students want to try advanced techniques that require a lot of data, and often the data doesn't exist; e.g. there may be insufficient data as far as time series are concerned. In these cases, allowing them to do some exploratory data analysis rather than formal modelling is sufficient. Finally, guidance on writing up their project reports is given. This is important, as the entire mark for the module is based on this assessed piece of work. 3.4 Case study 4: Applied Economics project with an Evidence-Based Approach: MA at the University of Manchester In this proposed taught Masters course, "Applied Economics Project: An Evidence-Based Approach", an empirical project will take the place of what otherwise may have been a more standard academic dissertation. This supports the overall thrust of this Masters course, which is about general employability rather than training for further postgraduate research in economics. The course will discuss how to formulate a research project, and introduce students to a range of data sources, types of data, software tools and applications, types of variable, and structure of data. From the start, students are made aware that they will be delivering, over the course of the year, an econometrically and data driven piece of work, rather than a theoretical dissertation. This cohort will have already done an additional unit on econometrics, so their underlying theory should be sound in any case. The employability requirement driving this course hangs on students proving their ability to identify a problem, produce a sound piece of analysis around it, and present this analysis clearly in a written report and a verbal presentation. As with the ADEP course (see case study 3 above), the course begins with lectures going over the econometrics ideas and techniques students will require, and the software packages and data sources (including ESDS International) that they might use. Lectures will also cover how to think about developing project ideas, how to use econometrics for their analysis and integrate this properly into their reports, and what is required from their verbal presentation. Tutorials for this course, unlike the ADEP course, will be in the format of small groups discussing their project plans together. Although the projects themselves are individual, these groups will work together throughout the course, ameliorating the problem of trying to support such a diverse array of problems under investigation. In general, exploratory data analysis will be all that is required from students in their reports. Some more advanced students, or those with project topics that are simple enough to allow it, may also be able to build a model. Examples of exploratory data analysis will be given, and, rather than IT lab tutorials, there will be lab clinics during the period when students are working on their analysis, so that those who need support with the practical aspects of data extraction, manipulation, analysis and presentation can drop in and request it. Students will present the results of their analysis through a verbal presentation and written report. The written report must explain the problem it seeks to address, and the results of the analysis, to an informed reader. It is expected to be the sort of task somebody working as an economic analyst might be asked to do. However, it also needs to be academically correct and rigorous. The lecturer for this course sees it as a journey through a complete work-related project driven by the employability requirement demanded by many students. The project report itself can be used to demonstrate the empirical data handling skills gained to prospective employers. 4 CASE STUDY OUTCOME: THE TEACHING TOOLS AREA OF THE ESDS INTERNATIONAL WEBSITE A huge amount of useful information was collected during the “Real World: Real Data: Real Stories project”, in the form of video footage, audio recordings, transcripts of interviews and a final report which includes recommendations for improving and further embedding the service in learning and teaching. Whilst it had originally been anticipated that the case studies would be presented as short narratives, the sheer amount and quality of material generated led to a new intention to produce a ‘Teaching Tools’ package with interactive embedded multimedia components in order to share these user experiences with the wider teaching and learning community. The Teaching Tools materials have been developed as an open access, stand-alone website linked to from the ESDS International website. They comprise of three sections; ‘Getting Started’, ‘Teaching Tools’ and ‘Hear from real lecturers’. The ‘Getting Started’ section contains an e-tutorial on registering to access the data, a printable guide to registering and FAQs. The case studies collected during the project have been used to produce ‘Sample Course Plans’ within the ‘Teaching Tools’ section of the materials. These include an overview of each course, i.e. teaching level, structure and contents, credits awarded and assessment method. Also included for each course are the aims, learning objectives and relevant links for each of the courses. Step by step guides, examples of student work, training information and links to other useful e-learning resources are also provided within the ‘Teaching Tools’ section. The ‘Hear from real lecturers’ section contains a video of lecturers talking about why it's so important to use 'real' data in teaching and their experience of using data from ESDS International in their courses. Not including the home page, the most popular pages are the: Guide to Registering; Sample Course Plans; followed by the Step by Step Guides. The resource has been designed for newer lecturers, initially in economics (though other disciplines will be included as the resource develops) who are unfamiliar with using real data in their teaching. By providing real life examples of the use of international data in teaching, the resource is intended to give educators the confidence to use real world data in their teaching and to help with embedding real world data into their courses. The Teaching Tools resource is available at: http://esds.ac.uk/International/elearning/teaching-tools/ 5 FINDINGS AND RESULTS The case studies revealed that utilising real-world data, extracted by the students themselves to teach econometrics and related skills, has a number of key benefits over using fictional or pre-configured datasets that have been developed purely to teach specific learning activities. These include: • • • • • • • Exposing students to how economics data looks in its raw state, and to the complexities and difficulties of working with such data. “I suppose it's connected with getting your hands dirty and looking at the real world, and lots of the standard courses have got data that's already in some senses cleaned and used to show an econometrically interesting point of view whereas when you actually start to investigate the world you often find that things aren't straightforward, and so estimating models often you find problems with significance and the lack of the assumptions being met.” (Nick Weaver, Co-Director of the Development Studies Stream and Programme Tutor on the BA Econ.; Teaching Fellow, BA Development Studies, PG Diploma Economic Development, MA Development Economics, University of Manchester, in interview 28 July 2009.) Operationalising real research questions in terms of available data. Identifying and locating data that can support or refute a hypothesis is a key real professional skill for students. Data for developing countries in particular is often messy and patchy but through the experience of handling real world data, students learn how to adjust their research questions or methodologies, or consider appropriate proxy variables. “A particular problem [...] is the fact that there is often insufficient data as far as perhaps the time series is concerned to show what people want to show and then just doing exploratory data analysis might be more important than formal modelling, and that's one big problem for development economists in general, that they often want to do advanced techniques that require a lot of data and there isn't that data there so actually exploring what data's there can actually help people realise that.” (Nick Weaver, in interview 28 July 2009.) Providing a crucible within which academic practice can be improved and a scientific approach to econometrics encouraged. By using real data which they have located and extracted themselves, students learn to correctly document and cite the data used, and the techniques used to manipulate it, in a way that makes their work more replicable. Giving teachers an opportunity to flexibly develop learning activities around a range of realistic, topical, relevant and interesting scenarios, and to allow students to select their own area of research. Giving students practical, generalisable statistical, numeracy, computer and information skills that are directly applicable in the workplace. Students gain skills through the practical experience of locating and accessing public sources of data. They can extract, then re-extract data any number of times. Mistakes can be rectified, and methods can be easily tried out against different variables. Ease of experimentation, without fear of losing data, is a good learning tool and builds student confidence in learning new software packages and confronting data for the first time. Forming a solid basis of skills for students’ future research, from undergraduate level dissertations through to post-graduate study and beyond. Giving students the opportunity to develop a portfolio of realistic project work that can be used to demonstrate skills and experience to employers. The employability benefits to students both in using real world data and learning real world skills for the workplace as a result were highlighted repeatedly by the interviewees. The value of free, easy-to-access, authoritative socioeconomic data resources was a key finding from these case studies. Teachers reported that having a single online place and system to access the data from was hugely beneficial, saving them time and effort in having to bring together multiple data sources. To this end the ESDS International website is also a valuable resource in its own right, providing information on the data, country and variable coverage, and examples of use in research and teaching. Teachers reported that students like the ability to be able to access multiple databases through a single interface, that if they lose their data they can re-extract it as many times as necessary, that it helps build their confidence in handling data – sometimes in large quantities, and especially that they can see the benefits to their future careers of using this data in their studies. Using software tools and packages to analyze the data was also seen as an opportunity to develop important skills. The case studies provided the opportunity to talk to teachers directly about the value of having teaching resources shared across the community. There was a positive response to this idea, with teachers reporting that firstly they would be willing to share their own resources with other teachers and secondly that they would find real benefit in having access to a pooled set of resources. This concurs both with the findings from the MacInnes report [4] and the utility of a national place for sharing teaching resources through a community of practice [5]. Notably, one of the case study teachers had already shared resources he had created through YouTube. This willingness to share their own, and use others’ materials, is at the heart of the Teaching Tools website. It speaks to the Open Educational Resources ‘(OER) movement’ that is embracing open sharing of resources in support of teaching and learning on an international scale; and positions ESDS International well to respond to initiatives being led by UNESCO [6] and others such as the Global Courseware Consortium. By emphasing better research practices which improve replicability and scientific literacy, teaching based on real world data also chimes with the wider Open Data and Open Knowledge agendas. Finally an unanticipated finding of these case studies was the valuable input that came from teachers sharing their pedagogic approaches. Not only were they happy to discuss how they used the data in teaching, but they wanted to share with us the approach they took to embedding this in the curriculum. As a data delivery service this is incredibly valuable information as it enables us to find how data can be integrated into the disciplines. Moreover it provides information about whether the service we are providing our ‘users’ meets their needs. As a user-led service organization this work has provided us with a valuable opportunity to develop our service and position it to be core to teaching with data. This is critical in order for us to be instrumental in providing a service that can assist in combating ‘the skills deficit in quantitative research methods across the social science research base’ [7] in the UK. The work presented in this paper positions ESDS International well for the renewed interest in the UK in statistical literacy, as evidenced through the Royal Statistical Society’s getstats campaign. As governments start to take seriously the ‘decline’ of statistics as a discipline [8], and introduce initiatives to respond to quantitative sciences as a vulnerable subject, the work presented in this paper can help us learn from best practice existing in quantitative methods teaching in economics, and consider how these methods can now be extended to other social science disciplines. REFERENCES [1] MacInnes, John “Proposals to support and improve the teaching of quantitative research methods at undergraduate level in the UK.” 2009. http://www.esrcsocietytoday.ac.uk/ESRCInfoCentre/Images/Final%20Report_%20Strategic %20Adviser%20for%20Quantitative%20Methods_tcm6-35465.pdf [2] Carter J. (2010) REAL DATA; REAL WORLD; REAL STORIES: A CASE STUDY APPROACH TO DEMONSTRATING IMPACT ON THE STUDENT EXPERIENCE, EDULEARN10 Proceedings, pp. 4792-4802 [3] Currier, S. “Mimas learning and teaching stories project: final report”. 2009. (Internal document) [4] MacInnes, op.cit. [5] Carter, J 2011, ‘Jorum: A National Service for Learning and Teaching’ in Teaching Quantitative Methods: Getting the Basics Right. Payne G and Williams M (eds), SAGE Publications, London [6] Open Educational Quality Initiative 2011, Beyond OER: Shifting Focus to Open Educational Practices, Government of Australia, Available from: http://portal.unesco.org/ci/en/ev.phpURL_ID=31243&URL_DO=DO_TOPIC&URL_SECTION=201.html [10 May 2011]. [7] ESRC Quantitative Methods Initiative, Available from: http://www.esrc.ac.uk/funding-andguidance/tools-and-resources/research-resources/initiatives/qmi.aspx. [19 May 2011] [8] Fearn, H 2011 ‘Statisticians worried by field's rate of decline’, Times Higher Education 10 February 2011 Available from . http://www.timeshighereducation.co.uk/story.asp? storyCode=415137&sectioncode=26 [19 May 2011].