Economics assessment and ChatGPT

At the end of last year OpenAI launched ChatGPT – Generative Pre-trained Transformer. It has the ability to impact traditional assessment methods by generating answers to questions which are often indistinguishable from a student response.

ChatGPT operates using algorithms that process data, allowing it to string words together in response to a prompt. Unlike humans, ChatGPT has access to vast troves of information available on the internet and uses large language modelling to recognise patterns in the words in each prompt to mimic human writing when dispensing knowledge.

A recent research paper entitled ‘ChatGPT has Aced the Test of Understanding in College Economics: Now What?’ by Wayne Geerling; G. Dirk Mateer; Jadrian Wooten; Nikhil Damodaran (2023), looks to evaluate if ChatGPT could outperform the average undergraduate student in economics using the Test of Understanding in College Economics (TUCE) which has been in use in US Universities for more than 50 years. There are two multiple-choice tests of 30 questions covering micro and macro economics.

The two tests were conducted with thousands of economics students from US universities and were sat before the start and at the end of the semester. The pre and post results would enable educators to measure the impact of particular pedagogy over this time period. The results were as follows with most students answer around 40–50% of questions correctly. The authors then put the two tests through ChatGPT and found that it answered 19 of 30 microeconomics questions correctly and 26 of 30 macroeconomics questions correctly, ranking in the 91st and 99th percentile, respectively – see graph for microeconomics test.

Some interesting findings regarding ChatGPT answers include:

• Choosing all 4 options as an answer to a multiple choice question
• Being unable to process images
• Questions answered wrong in the micro exam include: Supply and Demand x2, Factors of Production, Utility, Elasticity, Comparative Advantage, Externalities, Market Structure and it did not answer Profit Maximisation. Profit Maximisation not applicable
• Questions answered wrong in the macro exam include: Components of GDP, Tools of Monetary Policy x2, Exchange Rates.

Where to from here?
Using software tools such as Turnitin may not be sufficient to spot a student answer using ChatGPT therefore educators need look at designing assessments that focus on critical thinking and analytical skills that cannot be easily duplicated by AI.

• Assessments need to reward students that know the content and not those that are able to source answers through classmates or ChatGPT. By introducing time restrictions those students who have knowledge of the material are in a much better position to answer more questions.
• It is important to highlight that although ChatGPT looks to be very valid in its response to a question it doesn’t mean that it is correct. A popular recommendation amongst teachers is to produce ChatGPT with errors and have students to identify as many as they can.
• There are other ways to engage students in learning experiences that can’t be replicated through ChatGPT namely classroom presentations, data response type questions, in-class writing assignments, collaborative learning project with students in different countries, quizzes etc. This goes beyond the simple memorisation of notes and theory and addresses the complex nature of economics with a deeper understanding.
• Tools like ChatGPT are likely to become a common part of the writing process, just as calculators and computers have become essential tools for learning mathematics and science. The challenge of universities is to adapt their curriculum to this new reality and to embrace the new era with innovative and effective assessment strategies.

Source:

ChatGPT has Aced the Test of Understanding in College Economics: Now What?’ by Wayne Geerling; G. Dirk Mateer; Jadrian Wooten; Nikhil Damodaran (2023)

Use elearneconomics for immediate personalised feedback on Micro and Macro topics with tasks designed for true student-centred learning and understanding that improves students results and grades.

Using playing cards for economics discussions

No doubt you’ve had plenty of discussions with your classes on economic issues. One of the challenges is to keep students on task and try and get contributions from all students. In order to overcome these issues I have developed a set of playing cards with certain statements on each. Students receive 8 playing cards with different assessment objectives/ skills/ elements of written work in economics – see photo below.

An exam essay question is given to students and then they debate the questions amongst themselves. However students can only speak when they play a card and they must follow what the card says – e.g. Argument, Building on someone’s point etc. This limits each student’s number of responses and makes sure the discussion helps students practice for an assessment, according to the assessment objectives. It also allows for greater contributions from other members of the class. It generally works well although at times you may have to play one of your own cards to keep students on task. I have attached a link to a document with all the statements – below. All you have to do is buy some packs of playing cards and use wide sellotape to attach statements to the cards. Be interested to know how others get on.

CARDS

Synchronous vs. Asynchronous learning and the laundry test.

As we are back in lockdown in New Zealand I thought I would share a post that I did last year relating to the challenge of finding the right balance between synchronous and asynchronous material. My ‘Webex’ lessons were predominately asynchronous in that I wanted to get through material and also the fact that a lot of the more engaging aspects of my teaching are difficult to do through the Internet. Although you could do some engaging activities through chat forums nothing beats the energy and engaging nature of face-to-face in the classroom environment.

An article which I picked up from Michael Cameron’s blog ‘Sex, Drugs and Economics’ makes for very good reading.

Dan Levy ‘The Synchronous vs. Asynchronous Balancing Act’ Harvard Business Publishing Education. 7th August 2020.

Asynchronous learning is better when you think it is important to have the following:

• Students developing a common foundation before class (especially of basic ideas or concepts).
• An assessment of your students’ perspectives or background on the subject, as this will affect how live classes would be conducted.
• Students being able to engage with the material at their own pace. This is especially useful if prior knowledge of the material varies a lot across students.
• Students spending a substantial amount of time pondering and reflecting.

Synchronous learning is better when you think it is important to have the following:

• Exchanges of perspectives among your students.
• Students learning from each other.
• Interactions in which you’re playing the role of facilitator or mediator.
• Opportunities to build community.

Levy comes up with a novel way of looking at synchronous v asynchronous delivery.

Where I teach, online classes generally get recorded; students can watch the recorded videos if they cannot attend the live session. I recently asked a student how she decided whether to engage in the live class or watch the recording later. Her answer was revealing. She said, “When I am trying to decide, I ask myself, ‘Is this a class I could attend while folding my laundry?’ If the answer is yes, I watch the recording. If the answer is no, I attend the live session.”

While I think that, in general, we should design both synchronous and asynchronous experiences that students find so engaging that they cannot fold the laundry at the same time, I think the spirit of this question might help inform your decision of what to reserve for asynchronous learning. If the students can conceivably fold their laundry while engaging in the experience, my advice is to either eliminate it or reserve it for asynchronous learning.

As Cameron points out if a student could be folding laundry in your class you need to look at how you deliver your lessons / lectures. Class time is an opportunity to engage students in learning experiences and getting them to think for themselves. For this to work not only has the teacher got to have energy but the course / assessment at the end of the year has to encourage a type of thinking.

“Real thinking does not install knowledge in the brain: rather it evokes potential that exist in the student, developing innate talents and abilities.” Mind Over Water: Lessons on Life from the Art of Rowing by Craig Lambert 1999.

Rethinking Economics – Econocracy

You might have come across the book ‘Econocracy’ written by students from the University of Manchester which makes three big arguments about the relevance of economics courses at Universities.

First, economics is part of all aspects of our public life. Second, the economics profession sees the economy “as a distinct system that follows a particular, often mechanical logic” and believes this “can be managed using a scientific criteria”. It would not be recognised by Keynes or Marx or Adam Smith. Thirdly, the authors criticised what economics students are being assessed on – models or theories which were being memorised for exams.

The interview below on Newsnight (BBC2) has author and student Joe Earle and Professor Diane Coyle (follow her excellent blog The Enlightened Economist) discussing the state of the discipline at University and what they don’t teach to economics students. More information can be found on the website Rethinking Economics.

Economics website for IGCSE AS A2 and IB courses

Want to learn or need assistance with Economics? Are you studying or teaching A Level Economics, Advanced Placement, or International Baccalaureate (IB)?

Help is at hand, elearnEconomics assists individuals studying Economics. This site covers a wide range of courses and individuals have the ability to customise their course or do extension work. It’s simple, easy to use and very cost effective.

eLearnEconomics is a comprehensive online economics learning resource. It is for both students AND teachers. Students study the concepts of each topic with the key notes, then review those concepts with the audio/video and flash card sections and finally test themselves in the written answer and multi-choice sections. The multi-choice section records student scores enabling them to track their progress and build their confidence leading into exams.

Teachers have the ability to monitor students progess within the teachers’ administration section. Students can be arranged into class groups and full reports generated to quickly identify problem areas. These high quality PDF reports can also be presented at parent/teacher evenings. Click the link below to access the site.

elearneconomics

Using UB40 to teach Unemployment

I recently started teaching the Unemployment topic to my Year 11 IGCSE class and remembered that one of the first albums I bought was UB40 Signing Off – released in 1980 (see right). The front cover and reverse has been made to look like the UB40 unemployment benefit attendance card from which the band took their name. Their UK top-ten hit “One In Ten” was an attack on Thatcherism and is mistakenly cited as referring to the number of unemployed in the UK at that time. It is in fact a song about government statistics in general, and how politicians use them to de-humanise problems. Useful way to introduce the subject especially if the class like reggae. I found it useful to have two windows open and play the video along side the lyrics. Click here for the lyrics of the song and here to see UB40 perform on Top of the Pops in 1981.

Relevance of Economics Teaching at University

Here is a link to Peter Day’s Global Business programme from the BBC World Service – Economic Rebellion. In this episode he looks at the history of economics teaching and asks why some universities are changing their courses whilst others are staying put. Since the crash of 2008 students have been rebelling against the economics teaching. In an interview with John Kay, he quite rightly stresses the importance of teaching ‘Economic History’.

Arthur Marshall gave his advice to the economics profession.

* Are you covering different schools of economic thought in your teaching? Is your approach intellectually pluralistic enough?

* Are you teaching enough economic history, so that we can learn the lessons from the past?

* Do your students have enough time to absorb and reflect on the material they are learning? In fact, as the discipline expands, are your students taking enough economics?

* Are you encouraging your students to embrace and respect the perspectives that other disciplines bring to thinking about and solving economic problems? What, for example, have we learned about neuro, evolutionary and behavioural economics? And how can that learning be better incorporated into public policy-making?

* How can we better understand the trade-offs between policies that improve incomes and those that improve social inclusion or environmental sustainability or our resilience to economic shocks?

* And, perhaps most importantly, are you challenging yourselves, and your students, to think beyond the comfortable?

Having read ‘This Time is Different’ by Carmen Reinhart and Kenneth Rogoff you often wonder why economists and public officials didn’t pick the common patterns amongst so many previous financial crises

I can also recommend the book “What’s the use of Economics”, edited by Diane Coyle, which examines what economists need to bring to their jobs, and the way in which education in universities could be improved to fit graduates better for the real world.

The Relevance of Economics and Economics Teaching

Gabriel Makhlouf, Secretary to the Treasury (New Zealand), gave a very thoughtful presentation at the Government Economics Network Annual Conference in Wellington. The main focus of his talk was the relevance of economics and economics teaching. He recalled his early days studying economics at University when Lipsey was the textbook of the day and stagflation and the ascendency of Friedman over Keynes were the talking points.

He quite rightly criticises the reliance on mathematical models in decision-making and he suggests that problems arise when the straightforward models,designed to think in a structured way about the economic issues, are confused with the reality we are trying to address. That risks confusing a moral science for a natural one.

Although we can build economic models based on rational consumers with perfect knowledge there is a need to accommodate unintended events which are part of the imperfect market. As Paul Krugman put it, “economists, of all people, should have been on guard for the fallacy of misplaced concreteness” and “the economics profession went astray because economists, as a group, mistook beauty, clad in impressive-looking mathematics, for truth”.

He quotes Arthur Marshall

“I have a growing feeling that a mathematical theorem dealing with economic hypothesis is very unlikely to be good economics, and I go more and more on the rules:

1. Use mathematics as a shorthand language rather than as an engine of enquiry.
2. Keep to them until you have done.
3. Translate into English.
4. Then illustrate by examples that are important to real life.
5. Burn the mathematics.
6. If you can’t succeed in 4, burn 3. This last I do often.”

Also of note is his advice to the economics profession.

* Are you covering different schools of economic thought in your teaching? Is your approach intellectually pluralistic enough?

* Are you teaching enough economic history, so that we can learn the lessons from the past?

* Do your students have enough time to absorb and reflect on the material they are learning? In fact, as the discipline expands, are your students taking enough economics?

* Are you encouraging your students to embrace and respect the perspectives that other disciplines bring to thinking about and solving economic problems? What, for example, have we learned about neuro, evolutionary and behavioural economics? And how can that learning be better incorporated into public policy-making?

* How can we better understand the trade-offs between policies that improve incomes and those that improve social inclusion or environmental sustainability or our resilience to economic shocks?

* And, perhaps most importantly, are you challenging yourselves, and your students, to think beyond the comfortable?

Having read ‘This Time is Different’ by Carmen Reinhart and Kenneth Rogoff you often wonder why economists and public officials didn’t pick the common patterns amongst so many previous financial crises

Below is a link to the full article – a good read.

Economics: Teaching, Applying, Learning

I can also recommend the book “What’s the use of Economics”, edited by Diane Coyle, which examines what economists need to bring to their jobs, and the way in which education in universities could be improved to fit graduates better for the real world.

Economics of Crime

Wayne Geerling of La Trobe University in Melbourne has produced some great resources using a variety of media. Here is a video showing “The Economics of Everyday life” and it relates it to marginal cost and marginal benefit analysis. Worth a look.

Still E.C.O.N.

Last week I was at the University of Waikato Professional Development Day for secondary school economics teachers. As well as presentations they showed work done by undergraduate students. Starting in 2011, students in ECON100 Business Economics and the New Zealand Economy have been given the opportunity to complete a video project as part of the assessment for the paper. Below is the students’ Choice Award for Best Video:

The modern economy – what students should be learning.

Here is an interesting article from the New York Times. Gregory Mankiw – Harvard Economics Professor – outlines what is required to understand and be prepared for the modern economy. He comes up with 4 main areas of learning:

Learn some economics – As the economist Joan Robinson once noted, one purpose of studying economics is to avoid being fooled by economists.
Learn some statistics – One thing the modern computer age has given everyone is data
Learn some finance – Americans are increasingly in charge of their own financial future
Learn some psychology – A bit of psychology is a useful antidote to an excess of classical economics. It reveals flaws in human rationality, including your own.
Ignore advice as you see fit – The one certain thing about the future is that it is far from certain. I don’t know what emerging industries will be attracting college graduates four years from now, and neither does anyone else.

Maybe all of this can explain the graph on the left. Click here for the full article from The New York Times.

Museum of American Finance – New York City

There is a cool tour of this museum which has had a major revamp in the last few years – see below. Also their website has a variety of educational materials that might be useful for the classroom. Click here to go to their website. I can recommend the ‘Tracking the Credit Crisis’ poster set from the museum shop – great value and I have bought some for the department. See a small part of the poster set below.

Video Graphics from The Economist

The Economist has a wide variety of video graphics on its website. Topics include: the German Economy; FIFA World Cup; Oil in Louisiana; Banks and Risk. Very informative and great use of graphics. Click here to access the site.