SECTION 3- PROCEDURES

This section is aimed at describing the research methodology. It is divided into three parts which include; 1) explain the context of the case study; 2) design of action research process; and 3) describe the procedure used in designing the model of evaluation procedures.

This project will create a model for training evaluation for the Nursing Quality Development Unit, The General Hospital, Thailand as a case study. The Case Study is a research approach focused on understanding the dynamics that exist within a single setting (Eisenhardt, 1989). Fundamentally, the Case Study provides a thorough examination of an individual case to allow one to correctly gain insight of the occurrence from the thorough examination of a single case (Becker, 1970). Fidel (1984) asserted that the Case Study endeavours, on the one hand, to attain a detailed understanding of the phenomenon being studied but to also simultaneously develop more general conjectural statements as regards consistencies in the recognized phenomena.

The Context of Nursing Quality Development Unit, The General Hospital

The role of the Nursing Quality Development Unit, General Hospital includes the five dimensions of innovative practice in nursing: quality improvement, research and continuing professional education, risk management, monitoring, evaluation, and consultation.

Team members are three clinical supervisors and five clinical educators who are professionals in their fields and relish the prospect of interacting and serving as a resource to the whole organization. The task of the clinical supervisor includes an extensive range of activities such as:

• Working with registered nurses to learn new skills, finding new research and creating advanced tactics to meet the needs of a tasking healthcare environment; and
• Creating advanced educational experiences.

Nursing Quality Development Unit team members have a common objective to:

• Ensure nursing services offers the public an all-inclusive, consistent, evidence-based approach to clinical practice
• Encourage professional development and education of the nursing staffs
• Assess and manage nursing quality and practice
• Assess and manage nursing-sensitive outcomes
• Review existing proof and design nursing performance advancement and research projects (Nursing Quality Development Unit 2012).

Methodology

Action research or formative research will be utilized in formulating the evaluation model for the continuing registered nurses’ training programs. This kind of research is proposed for designing a model that will manage instructional processes via the reflective procedure of unbiased problem solving (Soultana & Stamatina, 2013; Reigeluth & Frick, 1999).

The Action research method agrees that “themes” must take the place of hypotheses. An organization researches on how to introduce a specific information into the system (Checkland & Holwell, 1998), Also, action research is viewed as a more enabling experience for the researcher as individual intuitions and essential evaluations of complex circumstances can lead to; including an appreciation of the role which researchers themselves have played (Elliott 1991). The investigator worked at the hospital which is a case study of this project for eight years as a registered nurse and clinical educator who was involved in the matters of continuing professional education and evaluation.  Of equal significance, action research is used in health care organizations owing to the fact that the procedure is based on maintaining the focus on the `real world’, as opposed to the controlled environments. It recognizes the issues which social actors are faced with daily, and tries to challenge complex organizational practices (Kelly & Simpson, 2001).

Design of action research process

Figure 2:

Process

The Action research procedure involves five stages and they are as follows:

Stage 1: Analytical context and the existing evaluation model.

In this stage, the investigator will examine the context and the model that is presently been employed for evaluation purposes in the ongoing training at The Police General Hospital, Thailand. As Heritage (2013) states, reviewing different sources and decisions on necessary skills or concepts of domain is relevant before deciding and concluding on how the advancement will be introduced. The steps followed in this stage are:

Review of the document.

The reason for doing this is to provide an overall impression of the hypothetical background as well as the associated experimental data for this program context as supported by Lorenc et al. (2014). The official papers that will be put under review include:

The program evaluation theory and

The existing evaluation model

Interview: 

In this stage, participants include every staff member of the Nursing Quality Development Unit, The Police General Hospital partaking in the training program, and the human resource management division. This method utilizes the reflection on the framework and experience. It additionally analyses facts prior to settling on the type of development model for future use (Bond, 2013).

Stage 2: Designing the model

Following the content analysis during which the data will aid in changing the needed evaluation model based on standards, criteria, and indicators. This stage encompasses:

Designing of an evaluation model: 

The training evaluation model will be developed by making use of the analysed data from the first stage.

Expert review:

Asking professionals in program evaluation to evaluate the first model, authenticate it, and provide feedbacks. This professional utilizes the formative evaluation to assess the training. Hence, the team can add a content professional, instructors or even a designer (Triantafillou et al., 2003).

Stage 3: Model testing

The first model draft will go through testing at the Nursing Quality Development Unit, where the supervisor will be tasked with evaluating the model and providing feedbacks.

Stage 4: Revised Model.

The new information gained from the reflection and from the feedback in the last step will be useful in terms of refining the model.

Stage 5: Validated and adjusted Model

Modules or factors will be authenticated and modified until the model is satisfactory and illustrative of the real model. There will be a professional appraisal conducted in a similar manner as in stage 2 and Personalized evaluation will likewise be conducted in the same manner as it was carried out in stage 3

SECTION 4- FINDINGS

The results for each stage of action research methodology are detailed in this section. The results given include the training evaluation model, logic model, program training logic model, standard and criteria for Nursing Quality Development Unit, and The General Hospital used to conduct the training programs for registered nurse.

Results (Phase 1: Some preliminary information)

Document review

When it comes to developing the training program evaluation model, there seems to be need for more preliminary information for both the existing model and the theory of training program evaluation. Alkin, Christie, and Vo (2012) claimed that evaluation is a process of gathering information. Some of the past program evaluation are inadequate since they fail to cover the extensive array of activities needing information (Alkin, Christie, & Vo, 2012). The main aim of taking such a step is to give an overview of the academic background of pertinent collected data so that the model of training program evaluation system can be correctly built.

The program evaluation theory

The purpose of theory in program evaluation is a significant but overlooked area in evaluation practice and theory. Program theory has a number of import function that can advance our capability to simplify from specific evaluations, add to social science theory, and reach agreement in evaluation planning (Bickman, 1987). In the current step, the theory of evaluation in the training program will be reviews to finding the suitable standard for model development. According to Phillips (1997), every case of evaluation and measurement is unique, the procedure and concept that works for one case may or may not work in another case. Techniques can be borrowed but the results cannot be borrowed (Phillips, 1997). Evaluation is conducted for numerous reasons. Generally, they can be classified into two groups, to advance training process and to determine whether or not to keep going (Phillips, 1997). Also, the exact reason include;  to decide whether a program meets the aim, to identify the strengths and weaknesses, to decide cost, benefit, and to collate data to help in making future program decisions (Stufflebeam & Coryn, 2014).  The theory of program evaluation is validating the model of evaluation’s ability to aid the readers in comprehending how and why a program works or fails to work. Weiss (1997) claimed that being aware of just the result, regardless of whether we know them with absolute certainty, does not give us sufficient information to inform program improvement or policy revision. Thus program evaluation has to get theory to be a framework and carry out the evaluation methodically.  Rogers, Petrosino, Huebner, & Hacsi (2000) stated that evaluation will offer considerably more reliable information when they address mechanisms that utilize theory-based evaluation. Moreover, Weiss (1997) claimed that evaluation is not a simple process and that we need credible theories as a direction from the basis.

The existing training evaluation model

Nursing Quality Development Unit has been conducting training for registered nurses since 2002. The two key reasons of these trainings are to meet the quality assurance standard. It is extensively believed that quality assurance is nowadays of foremost importance to providers of goods and services in the society particularly in the health sector (Strauss, 1978). Quality can be described as the degree of similarity between the purpose of education and the actually approved care as the result of education (Strauss, 1978). Also, nurses require a particular kind of extra training to assist them with the needed knowledge and skills which will empower them to ascertain and respond to the needs of patients (Parle, Maguire, & Heaven, 1997). Hence, numerous training programs have been carried out by the Nursing Quality Development Unit for registered nurses as regards the maintenance of nurse’s quality and in order to meet the standard of QA. Another reason is to help nurses to meet the education regulation of the Nursing and Midwifery Council which requires 50 hours of continuing nursing education for renewal of licensing every five years (Thailand Nursing and Midwifery, 2014). However, up till now, evaluations of these training programs are restricted to the end of training surveys and are usually fixated on recognizing the various problems of the programs. Stufflebeam and Coryn (2010) expounded on the fact that evaluation can offer complete and comprehensive information that measures the evaluation of the training program though the program evaluation needs to be consistent and standardized. Moreover, regular training program design have not adhered to logical steps in their design and development, for instance, training programs are focused on the interests and the need of individuals and not to meet the needs of the organization. Thus, an evaluation should be planned as a program design (Morrison et al., 2010).

Interview

In a couple of evaluations, the program theory has been extensively designed but the evaluator is based on a review of research literature on identical programs or pertinent causal mechanisms, through deliberations with key informants, through a review of program documentation or through observation of the program itself (Lipsey & Pollard, 1989). The investigator interviewed the supervisors of Nursing Quality Development Unit through telephone interviews (Skype) and used semi-structured questions, the summary of outcomes are as follows;

Lack of evaluation skills.

The evaluated toolkit and training programs are a relevant approach to developing nurses’ skills and knowledge. Ironically, the basic skills of evaluation are conduct evaluation; discovering and translating information, communicating with trainee are the same as the main skills of a decent evaluator (Posavac, 2015).  Every staff of Nursing Quality Development Unit are Registered Nurses, they lack basic evaluation knowledge and skills in evaluation and assessment.

Lack of support

This is also a problematic area and a tough one to handle. This is usually contingent upon the organizational culture (Hofstede, Neuijen, Ohayv, & Sanders, 1990). The valuation philosophy of hospital still needs to better the behaviour of policy makers and key management members who do not place importance on training evaluation. A superior evidence base on which to strengthen performance measurement policy is necessary, and new ideas should undergo thorough evaluation (Smith, 2009). This might help to proffer a few insights for points to include in any ‘training evaluation awareness’ campaign that the organization plans to run.

Not planning ahead and no appropriate tools.

The successful evaluations merely need painstaking planning, the aptitude to ask decent questions, and a rudimentary grasp of how to locate relevant answers (Guskey, 2002). For valuable results to make considerate, responsible decisions as regards expert development processes and effects, the facility needs to make efficient plans and tools to provide relevant information that they can use (Bushnell, 1990).

In these conditions, the training program evaluation carried out by Nursing Quality Development Unit requires the interactive form of evaluation. Evidence from the evaluation may determine future development of plans for closing the gap of need between hospital, nurses and evaluator from quality assurance institute. Furthermore, the suggestions from evaluation will additionally inform the organization for policymaking on the enhancement or adapt program, validate accountability, and rationalize funding. The results of the evaluation will likewise furnish the hospital, as program owners, with information for the program training development.

Results (Phase 2: Design a model and validate the model)

Designing evaluation model: 

The aim of this phase is to develop a training evaluation model that can guide evaluations of an extensive range of clinical and continuing professional development. The project’s ultimate aim is to assist the supervisors of the Nursing Quality Development Unit to accomplish more regularity and control over the hundreds of continual registered nurses’ training program for which it gives funding, and simultaneously, raise the ability of other training programs to carry out their own training evaluations.

After appraising the program evaluation theory as well as the main features of the current training evaluation model, this phase will clarify how the model was improved by the training program evaluation and then appraised by professionals. Evaluating training necessitates a methodical and thorough approach to decide whether the training is effective. This is a step-by-step debate of each component of the model. It involves modest valuations of participant satisfaction; evaluations of skills, knowledge, and attitudes learned; more difficult and challenging evaluations of behavioural and organizational modifications that may be attributable to the training experience; and the constantly disregarded job of objectively evaluating instructors  (Freeth, Hammick, Reeves, Koppel, & Barr, 2008).

Selected training evaluation model

There is an unending argument in the field of evaluation as regards which model is more ideal to enable the methods involved. Eseryel (2002) expounded the point that diverse approaches to evaluation of training deliberated in this paper show that the undertakings involved in evaluation of training are difficult and not regularly well defined. Since evaluation undertakings in training situations involve numerous objectives related to multiple levels, evaluation should perchance be seen as a combined activity between training designers, training managers, trainers, and probably many more.

The investigator chose  Phillips’ Evaluation Model  (1983) for creating the training evaluation model because Phillips’ Evaluation Model  facilitates decision makers to choose, from numerous options, the package that will best improve the general efficiency of a training program (Liong, 2005). According to Bushnell (1990), the Phillips’ Evaluation Model as a system-based model can easily discover if training programs are realizing their true objectives. It also allows them to distinguish the kinds of modifications they ought to effect in order to advance course design, content, and delivery. Perhaps most essentially, it notifies them on whether trainees really gained the necessary knowledge and skills (Bushnell, 1990). The Phillips’ Evaluation Model recognizes seven steps of evaluation which are defined in more details as follows:

Step 1 Collect pre-program data on performance and/or skill levels

Step2 Collect post-program data on performance and/or skill levels

Step 3 Isolate the effects of training from other positive and negative performance

           Influencers

Step 4 Convert the data into monetary value

Step 5 Compute the rates of delivering the training program

Step 6 Calculate the ROI (return on investment)

Step 7 Pinpoint and enumerate the intangible benefits.

Customizing and expanding on Phillips’ Evaluation Model 

Figure 3: Training evaluation Model (First Draft)

Step 1: Conduct needs analysis

A needs analysis for training is the initial step in designing a training and evaluation program.  A documented needs assessment also aids the evaluator in assessing the reason for the training program (Arthur Jr, Bennett Jr, Edens, & Bell, 2003).  The needs assessment gathers information from nurses on what training is required and the best approach to have it delivered.  A need assessment can also be used to discover whether a reputable training meets the needs of the hospital.  A training needs assessment assesses preferred or actual behaviour and includes such methods as surveys, interviews, observations, and document reviews (Gupta, 2011).

Step 2: Design Training Plan

Designing a training plan is the second important step in creating a training program.  The training plan can be viewed as a management plan for multiple different phases of training goals, management/administration, and evaluation (Thomas, 2003).  The training plan has to be included in the evaluation model as this plan should be accessible to the evaluator for a review.  In a newly commenced training, an evaluation plan ought to be developed as part of this phase (Garavan & O′ Cinneide, 1994). The point of origin for creating any training is to ascertain, articulate, and document the objectives of the organisation.  This aspect of the plan uses the outcomes of the needs assessment phase (Laird, Holton, & Naquin, 2003).   This aspect of the plan includes instituting and projecting the budgets of the training and resources needed to deliver and oversee the training.

Step 3: Develop and test the training syllabus

Creating a training syllabus takes time, skill, and dedication (Fink, 2013).  It is imperative that the aim of the training be openly acknowledged so that the training goals can be refined, explicit curriculum materials developed, and teaching methods and media designed (Romiszowski, 2016).  The training syllabus development phase joins together all the information developed in the first two phases—needs assessment and training plan.  This phase provides information as regards the identification of need, aim of the training and expected results, length of training, target population, sources of financing, and management plan (Rossi, Lipsey, & Freeman, 2003).

Step 4: Develop and evaluation plan for training

Evaluating the training and the trainers is a crucial, frequently overlooked, aspect of the training development effort.  An Evaluation plan is a guide on what you need to assess, what information is essential, and who you are evaluating for (Moore & Dutton, 1978).The evaluation plan should enumerate the main evaluation questions and the comprehensive monitoring questions that assist with answering the evaluation questions. This lets the Nursing Quality Development Unit to ascertain the information they have to gather, and how best to go about this. Contingent upon the facts of the training and evaluation plan, they can pinpoint the individuals in charge of various tasks, and timelines. The plan out to be easily understood by anyone partaking in the project at any time and be detailed on the occurrences as regards monitoring and evaluation (Tweddle, Robinson, Pocock, & Roy, 2012). It is likewise imperative to remember that there are several types of evaluation. An evaluation plan should preferably be completed at the planning stage of a project, prior to commencing the implementation phase. This will let the Nursing Quality Development Unit to plan beforehand and data collection tasks that you may have to embark on, including pre-intervention surveys.

Step 5: Conduct training

Following the testing and modifications of course materials, the next phase of the training development process involves implementing the course.  How often a course is delivered is determined by the availability of attendees, trainers, and funding.

Step 6: Conduct evaluation

The fundamental aspect of the training process is the gathering of data. It is prudent to utilize standardized data collection tools, which are proven to be effective in real life situations, and enhance these if needed to capitalize on data quality (Phillips, 1997). Besides, using experienced data collectors, giving data collectors training on a particular job or tool and/or overseeing data collection across numerous data collectors can additionally help to decrease partiality or inaccuracies in the  acquired data (Polkinghorne, 2005). Data collection is not essentially the only duty of evaluators; the staffs of unit can help gather the data on behalf of the evaluator (Bringle & Hatcher, 1996).

Step 7: Revising the Training

The final step in the model is revision of the training.  The evaluation will have enlightened the evaluators and major stakeholders on the thought of the participants regarding the training, what knowledge they gained, how their performance and/or knowledge transformed, and perhaps how the organization profited from the training (Russ-Eft & Preskill, 2009).  This information can then be used to adapt the training as required to make it more effective, more goals-oriented, or more suitable to its audience.  Revision of the training can include something as easy as including more breaks or a larger classroom, to providing regionally-based training facilities, to bring in completely new instructors or changing the entire curriculum.  It is essential that once the training is revised, the evaluation carries on so that the training can be continually updated and improved. (Bradley & Connors, 2007)

Expert review

A professional review involves a usability professional using his/her knowledge and expertise of testing the model. The professional will identify problems and suggest solutions to enhance usability (Brink & Morse, 1991). The professional employed by the Nursing Quality Development Unit to review the evaluation model is a person from the fields of evaluation, assessment and, nursing with extensive experience in nursing education and the highest level of understanding in assessment and evaluation.

Generally, the professional is supposed to possess skills and knowledge suitable to the areas of activities in which they are appointed to help evaluate (Berliner, 1986). The professional must additionally possess an advanced level of professional experience in the public or private sector in at least one of the following aspects or activities: research in the applicable scientific and technological fields; administration, management or evaluation of projects; technology transfer and innovation; use of the research results and technological development projects; and human resources (Jones, Armour, & Potrac, 2003). Additionally, the professional is expected to provide feedbacks and comments to the proposers (Blacker, Macchetto, Meylan, Stanghellini, & van der Marel, 2002).

It accepts that some professionals in a certain substantive region are adept at comprehensive examination and evaluation  model that could not be conducted through any other process (Shepard, 1993). The professional reviewed the evaluation model in figure 3 for validation and gave the following recommendation and feedback;

1) Appropriate design; the model kept to a logical step in their design and development based on evaluation theory.

2) The model match between evaluation methodology and Thai health care system.

3) As the Nursing Quality Development Unit have not used the evaluation model and evaluation before and the staffs of the Nursing Quality Development Unit lack evaluation and assessment skill, the model needs to include the workshop in order to provide skills and knowledge as well as direction to the staffs or supervisor.

4) According to the limited skill of the staffs, the example or the additional form or tools need to be provided as assistance to the staffs.

5) Cost and benefit in the model need to be discussed with the policy makers or the top management group for possible budget support for the training.

6) Before revising the model there need to discuss with the Nursing Quality Development Unit about the uncontrollable variable which can impact the outcome of the evaluation process.

Results (Phase 3: Test the model)

The first draft for the evaluation model in figure 3 have been tested at the Nursing Quality Development Unit, which used the One-to One evaluation method with three of the supervisors who were involved in the training and evaluated the training program. They evaluated the model and provided feedbacks as follows;

1. Step 1,2,3,5 and 7 will be conducted by the staff of the Nursing Quality Development Unit and the professor of the Partnership University.
2. In activity 3.2, and 3.3 the instructors who are the external source will provide the method and the media depend on the objective and on the instructors’ teaching plan and methods.
3. Step 4 and 6 are the major concern at the Nursing Quality Development Unit since it is a small organization, all staffs are nurses lacked evaluation know-how, the additional of explanation, example and packaging of tools for example criteria, standard, indicator , logic model and theory of change for the revised version.
4. In practical, the unit will have certain issues about upper management level staffs and head ward attitude as well as on the budget support of the evaluation. The investigator should provide the direction or guideline to solve or avoid these issues.
5. In case the staffs of the Nursing Quality Development unit cannot conduct an effective evaluation due to uncontrollable factors, the investigator should provide the option for readability.

Results (Phase 4: Refine Model)

From the professional and supervisor feedback and recommendations in step 3, the investigator has adjusted the model in step 3 and 4 as shown in figure 4 below;

Figure 4: Training and evaluation model (Revised)

The explanation in each step of developing the evaluation plan helps to assist the user with the details below;

Identify purpose of the evaluation

The two aspects of the evaluation serve as a basis for evaluation planning, focus, design, and interpretation and use of results. The aim of an evaluation impacts the identification of stakeholders for the evaluation, selection of specific evaluation questions, and the timing of evaluation activities. It is important that the program is clear about its intended objectives as regards the evaluation. Purpose priorities in the plan can help form the link between objectives and intended use of evaluation information (Rossi, Lipsey, & Freeman, 2003).

Develop evaluation Framework and logic model

An evaluation Framework also called a statement of theory, theory of program logic, or theory of program action  lays out the connections between the program strategy and tactics and the desired outcomes (Roehl, 2002). Evaluation offers overall general framework for evaluations across various programs or diverse evaluations of a single program for instance; output evaluation, and process evaluation. This includes program theory of change, program logic model and management processes (Funnell & Rogers, 2011).

1. Theory of change

In this study, the Kurt Lewin theory is employed as the theory of change in figure 5 shown below;

Theory of Change Model: Lewin’s Change Theory.

The Change Theory of Nursing was developed by Kurt Lewin (1947), who is recognized as the father of social psychology. This theory is his most influential theory. He theorized a three-stage model of change called unfreezing-change-refreeze model that needs prior learning in order to be rejected and replaced (Vioral, 2014). Numerous generations of management theorists have advanced this model, especially in relation to the possibilities for stability and an end to change. The relation between changes in professional practices and wide-ranging social change will not be linear or simple though the Kurt Lewin three-stage model gives an outline that helps us to visualize, plan and manage each of the stages of change (Cummings & Worley, 2014).

Program Evaluation – Program Logic Model