QUALITATIVE CASE STUDIES
Authors: Giel Ton, Karen Maas, Job Harms, Fédes van Rijn
This document outlines the methodology that will be used for the qualitative PRIME case studies. These case studies are complementary to the quantitative analysis of the PUM and CBI monitoring and evaluation data (see PRIME Policy Brief #3). As Yin (1984/2009, 2013) argues, a case study is named as such when it helps to answer a larger question. Therefore, case studies need to be sufficiently comparable and focused on this larger question. Case studies examine processes in specific contexts and are included as a research method when comparative or complementary analysis is needed.
The aim of the PRIME case studies is to provide qualitative insights into the impact of the activities of CBI and PUM on SME and BSO beneficiaries. Our research question for the case studies is ‘Why and how do CBI and PUM interventions work, for whom and under what conditions?’ This implies the need to study different types of beneficiaries and non-beneficiaries (e.g. larger and smaller SMEs) and different conditions (e.g. more and less competitive sectors). We therefore decided to focus on the breadth of the support in different sectors, assess the processes that generate impact in these sectors with the objective to learn from the mechanisms and processes that work in these instances to improve the effectiveness of CBI/PUM support.
Case studies will be conducted in at least six beneficiary countries in the sector where CBI and PUM are both supporting SMEs. The case studies will consist of in-depth and semi-structured interviews with client enterprises, non-client enterprises, BSO representatives, and local (sector) experts in order to get a deeper understanding of the mechanisms that affect SME processes and performance. The case studies will also consist of preparatory desk research, field missions and (comparative) analysis of the results. The case studies will include a cross-sectional survey to identify characteristics of supported and unsupported firms.
To gain insights in the effectiveness of CBI and PUM activities, we use qualitative research to identify the processes and dynamics that take place during and after the support trajectory and which influence the effectiveness of the support activities. This qualitative research will give us a better understanding of the context in which the interventions take place and an understanding of the characteristics of the support in the particular micro-context. Case studies thus help us to explore the mechanisms underlying the observed phenomena and changes.
After the first mission, the researchers will sketch (visually and in narratives) the intervention logic that is pursued in each sector/country, with the key assumption of impact, the intended and unintended effects, and the main enabling or constraining factors. This framework will be used later to organise the research synthesis and help us understand why the programme works differently for the various sub-groups, with different characteristics or under different conditions. The objective of the qualitative research is to identify, in 2017, the possibilities that will help refine this intervention logic.
We will include a sector survey in the case study methodology to collect basic information about the SMEs working in the sector. In doing so, the case studies can provide valuable insights about the differences between supported and unsupported firms. For example, CBI selects a certain type of companies that have potential for exports. Also PUM will reach the more innovative entrepreneurs in the sector. Through semi-structured interviews more insights will be obtained about the similarities and differences between the supported and unsupported firms in the sector.
Another important question in the PRIME research is if and how the support provided to certain SMEs might affect other SMEs, both negatively and positively. Indirect effects, along with positive and negative externalities, will be identified. An example of a negative externality could be that a supported SME obtains an increasing market share in detriment of other non-particpating SMEs. An example of a positive spillover would be that supported SMEs share the knowledge gained from the CBi and PUM support with non-participating SMEs in the region. The case studies provide an opportunity to identify these types of spillovers by interviewing not only the supported firms but by also reflecting on the dynamics in the sector with unsuported firms or sector experts.
In advance of each case study a preparatory study will be conducted. CBI and PUM activities in the respective country will be reviewed through the analysis of programme documents, the data on the supported firms in the last years, business case documents and personal interviews with CBI and PUM staff. The selection of the key sector in each country subject to PRIME research will be based on an assessment of the diversity in the support portfolio, synergy between CBI and PUM activities, and logistical considerations. Sectors with larger numbers of CBI and PUM beneficiaries are thus more likely to be selected. The feasibility studies are used as preparatory desk research and have to be approved by the Programme Board.
Selection of companies
We will select the SMEs for the qualitative interviews based on the potential to get additional insights about the processes, conditions and mechanisms that influence the effectiveness of CBI and PUM support. This implies a purposively selected sample of CBI and PUM supported firms; firms supported through similar programmes, and unsupported firms. The aim of the field mission is to visit and interview between 10 to 20 SMEs in the respective sector. Based on the first field studies in Bolivia and Indonesia in June 2014, the number of interviews is estimated to be feasible in a one-week mission,.
The selection will be made based on an overview of CBI and PUM participants of the last five years and the applicants to the new programmes starting in the selected countries. This provides us with an overview of the total sample of companies that have participated or are participating. Next to that, an indicative overview of non-participating companies will be made based on the available information provided before or during the mission (likely by the BSO and other experts working in the sector)..
After selection by the PRIME researchers, a local consultant will be hired to contact the chosen companies. CBI and PUM experts, local representatives, and country coordinators assist the consultant with making contact and in providing background information on the SMEs, but will not be present during the interviews. In most cases the interviews will be held with the SME contact person who is involved in CBI and PUM activities, and in most cases this will be one of the managers or the director of the company.
It is projected that each of the selected countries will be visited three times in the period 2014-2017. Whether a specific SME will be revisited during a subsequent mission will be decided upon case by case. This decision will depend on the potential of the specific SME to provide extra insights on the main research question: what works for whom under what conditions?
To enhance comparability of the case studies, we will use a common format for the semi-structured interviews as much as possible.
Preparation for interviews with SMEs will be done by reviewing the available information about the company using M&E data, intake form, exit form, the company website, Google searches, interviews with CBI and PUM experts, etc.
If possible and appropriate the interviews will be recorded. For each organisation interviewed, an interview report will be written with a level of detail that is sufficient for other researchers to use in the research synthesis to detect the enablers and barriers that influence the effectiveness of the support provided to SMEs.
Interviews and all the other information concerning the case study will be added to a portfolio document/folder with all rough data. This portfolio document will contain the information used for analysis and synthesis. The interviews will be processed (coded) in the qualitative software application Atlas.ti.
In terms of logistical considerations, and given the limited time available for each mission, it is important that supported firms, unsupported firms, and third parties in the respective sector, are somewhat clustered geographically.
Interviews with SME beneficiaries, SME non-beneficiaries, BSOs and sector experts about (i) firm selection, (ii) perceived impact of CBI/PUM services on firms, (iii) spillovers, and (iv) challenges and trend in SMEs and sector, will feed the answer to the question ‘why does it work or not work, for whom, and under what conditions?’.
A semi-structured questionnaire (see Appendix 1) will be used for the interviews with the various stakeholders (CBI/PUM experts, SMEs, BSOs and experts).
Each interview will take about 1–2 hours and will be recorded (for guidelines for the semi-structured interviews, see Appendix 2). The recordings will be used to gather quotes in subsequent reporting. If possible, we will also speak to other people in the firm and, if feasible, a company tour will be made as part of company visit.
It will be made clear to respondents that the information will be strictly used for research purposes and not shared with any third party without their explicit permission. Furthermore to avoid firms from feeling pressured to provide “desirable answers” or otherwise provide biased information, it will be explicitly made clear that the PRIME research team functions as independent researchers. For the same reason, if translators are needed for interviews, the PRIME research team will work with external parties that are not identified with CBI /PUM.
The critical success, fail factors and ideas for further improvement will be discussed during the interviews. We expect, at least, to get information on: the organisational challenges for the SME, the reasons for engaging (or not) with CBI/PUM, the ways in which they interact with CBI/PUM, and governance issues. Furthermore, we also intend to focus on SMEs’ relationship with their broader environment to gain a better understanding of how the CBI/PUM support might spill over to other firms.
Furthermore, interviews with the non-supported SMEs will be used to gain a better understanding of the differences and similarities between the two groups, as well as provide a better understanding of why firms make the decision to participate or not participate in the programmes. By repeated visits to the firms (e.g. a first mission in 2014 and a second mission in 2015) we might be able to identify several firms that decided to enrol in or withdraw from the programme during this period. We will then be able to explore in-depth about the mechanisms and processes that influence the decision of the SME to apply for the support.
The interviews with the BSOs and other relevant stakeholders in the sectors will be used to discuss how CBI and PUM support helped these organisations to improve their support to the SME sector. During the interviews with the BSOs we will also discuss possibilities to get access to the data of BSOs on non-supported SMEs, explore with them the diversity/heterogeneity of SMEs in the sector, and reflect on dynamics in the sector.
During the mission the need and potential for additional survey research will also be explored; for example, to quantify the qualitative insights about the characteristics of the sector, and between supported and non-supported firms. The survey will also be used to check the quality of the data submitted by SMEs to PUM and CBI; a check on the quality of the real-time monitoring data used in large-N analysis will be discussed with a local consultant.
Analysis of material
Each visit/interview will be processed in an individual report. Next to that a mission report will be made that describes the main results of all interviews in a specific country.
Each interview will be recorded. A report of one to three pages will be made for every interview. This will follow the actual flow of the conversation and not necessarily the semi-structured interview format. Interview reports will be put in a database, Atlas.ti, to distil common issues within and between each case study. Data collected from supported firms in the local survey will be compared to the data provided by them to CBI/PUM. Gaps between the two data sources will be identified and additional information needed and other interesting points for a potential follow-up visit will be defined.
After each mission a mission report of four to eight pages will be written. The mission report will provide information on sector level data and dynamics. It will provide an overview of the SMEs and other stakeholders visited. It will contain an analysis of the key observations of the interviews and per theme in the survey instrument. The mission report will be discussed with the research contact group and the involved CBI/PUM experts and be available on the website to show progress.
The first mission will capture the intervention logic of the support provided by CBI and PUM in each case-study. Follow-up missions may refine these intervention logics, disaggregating it into specific impact pathways for specific groups and under different conditions. These more refined impact pathways also serve to identify new themes and topics for additional research in the follow-up missions or in an additional survey.
We expect to collect useful insights with the case studies into how the activities of CBI and PUM unfold in the various regional and sector-specific environments that will be studied. These insights will in turn serve two main purposes. Firstly, they will enable CBI and PUM to further optimise their strategy and processes. Secondly, these insights can be used to optimize the M&E system, which in turn will enable for a better quantitative analysis of the impacts of the activities.
Finally, all data collected on one country/sector will be analysed in a case study report. The information from the database comprising all interview results will be analysed and main conclusions will be drawn.
The final pool of information, will be analysed using the software Atlas.ti (Friese 2014). In the synthesis process, we apply common codes to the information in each piece of information. During coding, the researchers have the option to write memos – associated with the quotations or documents, with observations, suggestions, ideas, etc, – that prepare for this final synthesis. These pieces of information are interview reports, project documents, secondary literature, CBI and PUM activity reports, export audits, etc. Whenever it helps to reflect on the barriers and enablers for effectiveness of the support, there is no restrictions in the type of document analysed in Atlas.ti..
To identify overarching issues and/or identify contradictory evidence, the quotations in the documents that respond to a specific code can be analysed separately. The output of Atlas.ti will be interpreted in the final case study report, with reference to the supporting evidence. The final synthesis report will have the following (tentative) headings:
- How do PUM and CBI clients compare to other SMEs in the various countries/sectors?
- What do companies and experts perceive to be the outcomes of the CBI and PUM interventions and for what type of companies, and under what type of conditions are they more effective?
- What do companies and experts perceive to be the main determinants of the success or failure of the CBI and PUM support activities?
- What do companies and experts perceive to the main possibilities to improve the CBI and PUM interventions?
Friese, S. (2014) Qualitative data analysis with ATLAS. ti, Sage.
Yin, R. K. (1984/2009) Case Study Research: Design and Methods, Fourth edition, Sage.
Yin, R. K. (2013) ‘Validity and generalization in future case study evaluations’, Evaluation, 19(3), 321-332.