In the second and third quarter of 2015 PRIME designed, implemented and analysed various online surveys. Moreover, the case study interviews have been processed with the qualitative software (Atlas.ti). Additionally, the PRIME research team developed the research methodologies for PRIME-ITC, which will be used for the impact evaluations in Kenya and Bangladesh.
Download the pdf: PRIME Newsletter 2015_Q23-final
Several online surveys were prepared and launched in support of the cohort analyses. We use it to collect data, which is not yet included in the regular M&E systems of CBI and PUM. The 15-minute online surveys provides the first cohort of data on supported companies, allowing us to collect data from SMEs and PUM/CBI experts and enables quality control. The online surveys were sent using the web-based survey application Qualtrics. August 2015 saw the launch of the PUM survey to firms. The surveys to experts will be implemented in Q4.
The PRIME research team analysed the data from the online survey sent to CBI-supported SMEs in Q4 of 2014. An online survey was sent to all SMEs participating in the integrated programmes of CBI; approximately 900 SMEs. Approximately 30% of these SMEs filled out the survey. This response rate is higher than expected, although we should look for ways of increasing the rate in the future. The analysis indicates that respondent SMEs are similar to non-respondent SMEs in terms of average audit scores. However, responding SMEs are on average smaller and response rates differ by geographical location. Surprisingly, response is especially higher in sub-Saharan Africa, and lower in East Asia. These issues deserve more attention in future analyses and efforts to increase response rates should be made. Most importantly, based on our preliminary analyses the quality of the information seems high; respondents left few questions unanswered and gave few “don’t know” answers. Moreover, the variation in answers to self-reported knowledge and practice questions, the self-reported changes in these knowledge and practice areas and the influence of CBI to these changes, shows that questions are likely to have been well understood by the respondents.
An online survey was sent to ± 2,800 PUM clients. In addition to questions on self-reported knowledge and practice, the survey also included questions about various business practices. These practice indicators serve to help us better understand how increased business knowledge drives improvements in SME performance. Furthermore, the research team tested various incentives to increase response rates and the response quality for PUM. The research team and PUM randomly allocated SMEs into three groups. The first group had the chance to receive a gift; the second group had the chance to receive a free expert visit (both based on a lottery system); and the third group received no incentive. The incentives were only valid when SMEs completed the online survey within three weeks of the launch. The rationale behind the experiment was that the most efficient incentive(s) (in terms of enhancing response rate and quality) can be used by PUM in the future for online data collection. The response after three weeks was 17%. Unfortunately, we found no major differences between the different incentives in terms of response rate (19% free expert visit, 17% smart phone, 16% no incentive). Three weeks after the survey was implemented, PUM made an effort to increase the response rate by having their local representatives urge and assist SMEs to fill out the survey. The local representative were put to work to increase response rates further; at this moment in time 24% of SMEs have filled out the survey.
A first coding tool was used for the analyses of case study material in Atlas.ti. The case studies should help us to identify the enablers or barriers to effectiveness of PUM and CBI support. The information, such as interviews, policy documents, mission reports, etc., are incorporated in Atlas.ti and coded according to a first coding structure. The case study analysis is an iterative process with new information being added to the pool of data each year. This “grounded” approach was presented and positively received during the Advisory Committee in June.
Country-specific impact pathways were developed to facilitate the case studies. Based on the feasibility report, the first case study mission and the debriefing with PUM and CBI staff, a country-specific intervention logic was developed (see the figure for Bolivia) and used to focus additional data collection. It showed the need to gain more insight into the characteristics of supported companies versus unsupported companies. For each country, similar focus areas are determined for the additional data collection (see arrows). Additional data collection will start in Q4 of 2015. This data collection will also be used for data verification: the research team will ask some questions that are also used in M&E (or the online survey) and compare the answers.
Detailed research methodologies for PRIME-ITC are under development. In PRIME-ITC we will evaluate the activities of the third phase of the Netherlands Trust Fund Export Sector Competitiveness Programme (NTF III – ESCP). NTF III Kenya’s main objective is to enhance export competiveness in the avocado sector, thereby improving the livelihoods of small-scale avocado producers. The rigorous impact evaluation in Kenya will follow a so-called “matched difference-in-difference” design in a sample of avocado farm-households that were supported (directly or indirectly) in quality improvement to meet the export requirements and a comparable group of non-supported farm-households. To support claims at the level of the farmer, the project will also look into changes at the level of farmer organisations and SMEs. The main objective of NTF III Bangladesh is to improve the export competitiveness of the Information Technology (IT) and IT-enabled services (ITES) industry in order to increase export revenues. PRIME ITC Bangladesh will undertake surveys to identify whether ITC reaches the SMEs that are most likely to respond positively to the support, show how the support affects the knowledge and practices of SMEs, and verify whether NTF III enhances the export competitiveness of the IT and ITES industry.
On September 7th the PRIME research team was invited to PUM’s offices in The Hague to provide input for PUM’s strategic planning for the next five years. During this session the research team presented its reflections on the proposed theory-of-change for PUM. The research team presented emerging findings from the various country-sector case studies and from a recent online survey with PUM clients. These findings were linked to the various assumptions underlying the PUM theory-of-change, including assumptions such as (i) that PUM support is additional to the local consulting market and (ii) that PUM’s selection procedures attract the SMEs that benefit most from PUM support. The feedback and reflections were recieved positively, as were various suggestions by PUM to the research team regarding the conceptualization of the various steps in the theory-of-change.