Measuring the effects of nursing education
After reading this article, you will be able to:
- Identify the four levels of evaluation
Years ago, while working in nursing professional development and education, consultant Gen Guanci, MEd, RN-BC, CCRN, realized that she was doing herself and her department a great disservice by reporting out “productivity” of the nursing education department.
At the time, Guanci was working on reports based on quantitative data—the department conducted 20 classes, for example, serving 200 nurses. (She calls this concept “butts in the seat reporting.”)
What they were not doing, she explains, was demonstrating what outcomes those filled seats then led to. “In other words, what was different as a result of our educational activities?” says Guanci, who is now a consultant with Creative Health Care Management in Minneapolis.
She undertook the task of identifying and explaining that qualitative aspect.
These outcomes were then linked to the organizations’ goals and even pay-for-performance initiatives.
“Many of these are stretch goals or outcomes some educators have a hard time relating their work to,” explains Guanci.
For example, let’s say your department holds education classes on computerized physician order entry (CPOE). One of the main reasons organizations implement CPOE is to reduce transcription errors. After your classes, the order transcription error rate drops by 66%. This helps validate the critical importance of nursing professional development’s role in regards to patient safety and outcomes.
The reason Guanci changed her view on how to demonstrate effectiveness of nursing education comes from her previous organization’s experiences on the ANCC Magnet Recognition Program® journey and its pursuit of the Baldrige Award.
“It’s not about how many people are in the seats—it’s about results,” says Guanci.
She advocates the use of Professor Donald Kirkpatrick’s Four Levels of Evaluation.
Level two looks at what new knowledge has been retained by the student.
Guanci felt that pursuing recognition for the results of the department’s work is something every education department has to be aware of. “I went this way originally because in times of economic challenge, education departments are often the first to be slashed and burned,” she says. “Leadership often doesn’t perceive the value the department provides.”
In her department at the time, Guanci was creating an outcomes report every six months—and having a terrible time getting credit for the work the department did. She knew she had to alter the way the department’s work was reported. Since then, the change has been notable.
“The process still occurs there,” says Guanci. “They’ve added positions instead of cut them—and [the education department has] really been able to create proof of worth for their department.”
In her previous organization, the education department felt it had sufficient evidence to show that as a result of foundational education it provided, it was able to assist in a decrease in transcription error rates.
“Educators have a hard time trying to take credit for things that change in an organization that start with their education,” Guanci says. “We know education alone doesn’t invoke change. It’s a combination of many factors.”
There’s a partnership that needs to be formed—educators provide the education, and then managers make sure improved performance occurs after the learning.
“You’re not saying that it’s only because of your work, but it was the foundational behaviors leading to future behaviors,” says Guanci.
There is also the matter of making sure the education department chooses appropriate targets. The system in which this concept was implemented was set up so Guanci’s department could access all necessary outcomes data. This came to bear when they were targeting areas for education. For example, a nursing director called and reported an increase in errors whenever nurses used a pain pump. The director then asked that the entire department be educated on pain pumps.
Before making a decision, Guanci and her team drilled down to determine the cause of the errors. They discovered that there had indeed been a spike in pain pump errors. However, they were low in number (three), and each error had occurred on one particular nursing unit. Looking deeper, it turned out they occurred on the same shift, and finally they discovered the errors were the result of one nurse who needed additional training.
“I made the decision that we were not going to educate the entire hospital on this matter,” says Guanci. “It wasn’t a hospitalwide problem.”
This is the department’s mind-set. Always look to the data and hunt for cause and effect.
“This is huge,” says Guanci. “It’s something [education departments] have often never been asked to do before.”
At national seminars Guanci has spoken at, she has found the topic to be “a bugaboo”—people are asking the wrong questions. “I’ll hear the question, ‘How are you measuring your hours per patient day?’ Education shouldn’t be measured in patient day!” she says.
Measurement gurus often try to slip education into measurement like any other measure of RN productivity. However, education is as much an art as it is a science in terms of measurability.
“Sometimes you’ll hear a department automatically jump to education—for example, let’s have a class for customer service,” says Guanci. “Educators will put together a customer service class. Then the original requester comes to you and says, ‘But they still are engaging in the same problematic behavior!’ It really is a matter of putting forward the mind-set of what do you want to see happen as a result of this education before you even plan the program.”
You have to define it before you can achieve it.
Another challenge: Quantifying evaluations
An evaluation might ask, “Did the program meet its objectives?” The answer might be yes, all of the objectives defined in the program were met. But were they put into practice after the program was over?
“The hardest part is educating the educators on how to write an outcome,” says Guanci. “I would ask for outcomes and I’d see four CPR classes with 22 attendees. That’s not an outcome! We have to step back.”
You can’t evoke these kinds of changes alone. It requires the entire department to understand what an outcome really is and hold fast to that belief.
The truth is, just because you told students something doesn’t mean you educated them—and just because you trained them doesn’t mean they’re doing it.
Don’t be afraid to let leadership see what the education department is doing. “You’re having an effect on outcomes in the organization, so claim it,” says Guanci.
HCPro’s Advisor to the ANCC Magnet Recognition Program®, October 2010, HCPro, Inc.
What are the Four Levels of Evaluation?
The Four Levels of Evaluation were first published by Donald Kirkpatrick, professor emeritus at the University of Wisconsin and a past president of the American Society for Training and Development, in 1959.
The four levels of Kirkpatrick’s evaluation model essentially measure:
- Reaction of student: What the student thought and felt about the training
- Learning: The resulting increase in knowledge or capability
- Behavior: Extent of behavior and capability improvement and implementation/application
- Results: The effects on the business or environment resulting from the trainee’s performance
New campaign focuses on CAUTI reduction
‘Take Every PreCAUTIon’ provides hospitals with resources and tools to prevent infections
After reading this article, you will be able to:
- Identify why CAUTIs are common in hospitals
- Evaluate ways to prevent CAUTIs
- Explain the cost savings of new technology such as antimicrobial catheters
Catheter-associated urinary tract infections (CAUTI) are the most common hospital-acquired infections (HAI). In fact, CAUTIs account for approximately 36% of HAIs, according to a March/April 2007 Public Health Report. CAUTIs are also associated with high costs, high lengths of stay, and unnecessary antimicrobial use, which can lead to further complications.
Prevention of CAUTIs requires the appropriate techniques—namely, don’t use a catheter if the patient doesn’t need one—and technology. Enter the “Take Every PreCAUTIon” campaign launched by Bard Medical at the June Association for Professionals in Infection Control (APIC) conference in New Orleans.
The campaign aims to provide healthcare professionals with evidence-based techniques and technology that can help reduce this common but easily preventable infection. The program offers free, easy-to-use, customizable tools that can be used to establish policies and procedures to ensure consistent practices among healthcare professionals.
Additionally, as many hospitals already know, CMS announced in 2008 that it would no longer reimburse hospitals for the additional costs associated with CAUTIs. This has drawn a lot of attention to the infection, particularly in the C-suite, says Jaime Ritter, MPH, CIC, clinical manager at Bard Medical in Covington, GA.
“Prior to [the CMS ruling], I think generally people felt that UTIs were kind of inevitable; if you put a Foley catheter in and left it in long enough, eventually a urinary tract infection would occur,” Ritter says. “And lucky or unlucky for CMS participants, most UTIs occur in patients over the age of 65, so most of them fall into the Medicare payment system. Prior to October of 2008, the hospital used to be able to add a comorbidity code to potentially get reimbursement. Well, that’s not the case anymore, so I think that really spurred a lot of people to look at UTIs more seriously.”
Ensuring compliance with tools
The simplest way to prevent a CAUTI is to avoid using a Foley catheter, which is the main source of infection, particularly when it’s left in too long.
But vowing not to use Foley catheters is easier said than done. Some patients require Foley catheters, but the best way to ensure compliance is to have a clear-cut policy that ensures staff members understand the appropriate steps to take when evaluating a patient.
“It’s hard when you look at UTIs because I think that a lot of infection preventionists say, ‘Well, we’re just going to get the Foley catheters out early,’ or ‘We’re just going to do better maintenance once they are inserted; we are going to make sure they are secure,’ or ‘We’re just going to stop using Foley catheters altogether,’ ” says Ritter. “None of that is really completely realistic, and it takes a lot of resources to do any of those things. So when we put together the campaign, we wanted something that would offer the infection preventionists a series of tools. The first is to assess the program: ‘What do I actually have, what do I need to put in place, and what can I improve on?’ ”
Once you have a clear-cut policy and procedure, you’ll need to periodically educate staff on following the policy. The “Take Every PreCAUTIon” campaign offers surveys and daily assessments.
It’s also important to empower nurses to follow the procedures correctly. Many nurses haven’t had any training on inserting a Foley catheter since nursing school, Ritter says, so supplying them with the appropriate equipment and establishing a strict procedure helps the process run smother.
“Nobody goes back and reminds people about Foley catheters,” Ritter says. “Foley catheters are very similar to any other procedure in healthcare, which is ‘see one, do one, teach one, and now you’re a master of it.’ ”
Combining HICPAC and APIC guidelines
Many facilities have already implemented APIC’s Guide to the Elimination of Catheter-Associated Urinary Tract Infections (CAUTIs), and many have recently begun using the Healthcare Infection Control Practices Advisory Committee’s (HICPAC) Guideline for Prevention of Catheter-Associated Urinary Tract Infections, 2009. Both provide guidelines and best practices for preventing CAUTIs.
The “Take Every PreCAUTIon” campaign combines the practical approach of the APIC guidelines with the evidence-based practices of the HICPAC document, according to Ritter.
“When we look at what policies and procedures you have in place, we took those things from the HICPAC documents, such as which patients should get a Foley catheter, and then the things that are suggested in the APIC elimination guide, which talk more about empowerment for early catheter removal,” Ritter says. “So we’ve tried to combine those two things to make them applicable for really any healthcare facility that may pick up the package and try and implement it.”
The cost associated with better technology
Part of Bard Medical’s “Take Every PreCAUTIon” campaign includes pushing hospitals toward better catheter technology, including silver alloy catheters with hydrogel coating or antimicrobial catheters. This is a 1B recommendation in the HICPAC guidelines, which state: “If the CAUTI rate is not decreasing after implementing a comprehensive strategy to reduce rates of CAUTI, consider using antimicrobial/antiseptic-impregnated catheters.”
As always, hospitals struggle with the cost associated with these products, and nurses often face even more of a struggle proving to administration that they are worth the money.
The “Take Every PreCAUTIon” campaign offers a very basic CAUTI tray that includes essentials such as hand sanitizer and wipes to clean patients before prepping them for a Foley catheter. Also included in the tray are a securing device and a checklist with questions for the nurse to consider, as well as a chart sticker that serves as a visual reminder that the patient has a Foley catheter. The cost associated with this new tray is only 25 cents more per tray, says Ritter. But the silver alloy–coated catheters end up being $5 more.
In the past, Bard has worked with staff to perform cost analysis by doing housewide surveillance and comparing the cost associated with new technology with the cost savings of preventing infections. “Now, putting in ‘Take Every PreCAUTIon,’ we’ll also be able to add in—as we move forward—the cost associated with reduced Foley catheters because one of the things we want to make sure is that we don’t overuse Foley catheters,” Ritter says. “If you are going to use a Foley catheter and the patient needs it, then we believe you should use something that is proven to reduce infections, but if the patient doesn’t need a catheter, then you shouldn’t use it at all.”
Briefings on Infection Control, September 2010, HCPro, Inc.
Study shows surgical teams still at risk for accidental sharps injuries
It’s time for hospitals to adopt preventive measures to protect all the members of surgical teams performing procedures in their operating rooms (OR), says California surgeon Ramon Berguer, MD, FACS.
That call for action follows a new study that shows the number of surgeons, residents, and nurses experiencing accidental injuries from sutures, scalpels, syringes, and other devices in the OR has actually increased by 6.5% since the introduction of the Needlestick Safety and Prevention Act in 2000. Berguer, chief of surgery at Contra Costa Regional Medical Center in Martinez, CA, and a lead author of the study, says he is not surprised by the findings.
“I work in the operating room; I know what the culture is like,” says Berguer. “I have adopted safety strategies myself and I have pushed my hospital to do so. I understand the struggle that is taking place.”
For instance, although statistics show that blunt-tip suture needles help reduce injury rates for every staff member participating in a surgery, many surgeons still won’t use them.
This is why Berguer argues that it’s time for hospitals to change their thinking and adopt policies to protect everyone in the OR.
The study, published in the April Journal of the American College of Surgeons, found that many surgical teams and hospitals are not using devices and procedures proven to reduce the risk of accidental needlesticks in the OR, such as blunt-tip suture needles and sheath scalpels. You can find the study, “Increase in Sharps Injuries in Surgical Settings Versus Nonsurgical Settings After Passage of National Needlestick Legislation,” at www.journalacs.org/article/S1072-7515(09)01654-8/abstract.
It’s not just surgeons who are at risk from accidental exposure to bloodborne pathogens from sharps injuries, but all members of the surgical team, says Berguer, a spokesperson for Ethicon, the major manufacturer and seller of blunt-tip needles in the United States.
Use of blunt-tip suture needles—which have been specially engineered to reduce the risk of accidental needlesticks and meet the standards under the Needlestick Safety and Prevention Act—have been met with resistance from some surgeons. Although they can’t be used in all instances, surgeons can use blunt-tip suture needles for suturing soft tissues, such as muscle.
What the study found
The study looked at data from 87 hospitals in the United States. Researchers analyzed findings from more than 31,000 reported accidental sticks from 1993 to 2006, including 7,186 sticks reported in OR settings. According to the study, most injuries were caused by:
- Suture needles (43.3%)
- Scalpel blades (17%)
- Syringes (12%)
The study also found that 75% of accidental sticks in the OR occur when medical devices are in use or are passed from one hospital worker to another. Nurses and surgical technicians are the ones typically injured by devices originally used by surgeons and residents.
More than 384,000 healthcare workers in the United States suffer needlestick injuries each year, putting them at risk for hepatitis, HIV, and other serious diseases.
The failure to reduce sharps injuries in the OR is a result of hospitals’ reluctance to adopt safety devices and strategies that are proven to work, Berguer says.
Part of the reason is the culture of the OR, where surgeons are in charge and there is an acceptance of risk. And up until recently, some safety devices have not been widely available or prominently marketed, he says.
There were also problems with some of the earlier safety devices.
“The first generation of sheath scalpels were both ineffective as scalpels and poorly constructed as safety devices,” Berguer says.
Further, there may not be an accurate picture of just how problematic the issue of sharps injuries is in ORs. Most needlesticks go unreported, Berguer says, as it can be a laborious process to report a sharps injury.
“For all these reasons, we’ve made little progress in the OR,” he says.
Ready for change?
Berguer says he hopes all that will change. “Right now we’re actually at a point where we can make an impact on needlesticks in the OR,” he says.
Surgeons now have the appropriate data to prove that sharps injuries in the OR are problematic, Berguer says. The industry has also progressed, adding better safety devices to the market.
OSHA’s Bloodborne Pathogens standard requires the use of devices to reduce needlestick injuries. However, OSHA does qualify its safe needle mandates by giving leeway to surgeons in its guidance documents.
For instance, surgeons don’t have to use a safer device if it “compromises either patient safety or medical integrity,” OSHA indicated in its 2007 bulletin developed with the National Institute for Occupational Safety and Health, titled “Use of Blunt-Tip Suture Needles to Decrease Percutaneous Injuries to Surgical Personnel.”
However, studies like Berguer’s now make it clear surgeons aren’t the only ones at risk.
“I think what’s been missing in the discussion is the fact that the needlestick risk is a shared risk among the team,” Berguer says. A decision made by a surgeon to use a sharp needle, for example, puts the entire surgical team at risk.
“I think that changes the tone of the discussion because now it becomes an employee safety issue,” says Berguer.
What happens next?
To bring about change in the OR, Berguer says he is in favor of ORs adopting a sharps policy that mandates the use of four strategies listed in the American College of Surgeons’ (ACS) statement on sharps safety:
- The universal adoption of the double glove technique or use of specially designed undergloves to reduce body fluid exposure caused by glove tears and sharps injuries
- The universal adoption of blunt-tip suture needles for the closure of fascia and muscle
- The use of the hands-free technique, which requires the surgical team to designate a sharps neutral zone for the pickup and release of surgical sharps, thus eliminating the direct handling of instruments from scrub person to surgeon and back
- The use of engineering sharps injury prevention mechanical devices as an adjunctive safety measure
Like OSHA, the ACS recommendations allow for discretion by the surgeon if circumstances could compromise safety. You can find the full statement at www.facs.org/fellows_info/statements/st-58.html.
“I think the strategy has been to try and have surgeons adopt them voluntarily, and that just hasn’t worked,” Berguer says. “But the fact that it is a shared risk puts the responsibility now on the hospital to protect its employees regardless of what the surgeon decides to do.”
“I think implementing a policy in the OR is the way to move this forward,” he says, and this needs to come from a hospital’s OR leadership group.
Currently, Berguer is working with several groups—the ACS, the Association of periOperative Registered Nurses, and the Council on Surgical and Perioperative Safety—to develop a template for a sharps policy that hospitals can adapt for their own institutions.
Safety devices are more expensive but save you the cost of sharps injuries
It may be more expensive for hospitals to use safer sharps devices in their ORs, but facilities will ultimately save money by avoiding injuries to members of their surgical teams, says Berguer.
Although safety devices can be more expensive, hospitals will save on costs associated with sharps injuries, says Berguer. Studies that have looked at the increased cost of safety needles versus the cost of reporting and treating a needlestick show a significant cost savings for hospitals, he says.
For example, blunt-tip surgical needles cost about 20% more than other sharp needles, Berguer says. However, each needlestick injury costs about $700 to report and $3,000 to treat with antibiotics. There is an additional expense if a hospital worker is injured and files a workers’ compensation claim. Needlestick injuries are the most common cause of accepted workers’ compensation claims in U.S. hospitals, he says.
“I think the C-suite can look at the numbers and definitely see these are substantial cost savings even with the slightly higher unit cost,” Berguer says.
Hospital Safety Center, August 2010, HCPro, Inc.
Multidose vials: Follow the 28-day rule
Multidose vials run the risk for infections and citations if they aren’t used appropriately
After reading this article, you will be able to:
- Define the 28-day rule for multidose vials
- Demonstrate ways to document compliance
On one hand, multidose vials are a great way to get more for your money with expensive medications. On the other hand, they can cause multiple complications if staff members are not following very precise procedures.
One of the primary concerns of multidose vials is the fact that multiple staff members enter the vial with a syringe, creating multiple opportunities to spread infection, says Peggy Prinz-Luebbert, MS, MT(ASCP), CIC, CHSP, owner and consultant for Healthcare Interventions, Inc., in Omaha, NE. In fact, the best solution for multidose vial complications is not to use them.
“A lot of the risk occurs when you go in and out of that vial,” Luebbert says.
But healthcare facilities don’t always have that option, and as a result they may have multiple multidose vials available for clinicians. This brings up the second problem: expiration dates.
Multidose vials have a limit on how long they can be stored after being opened or punctured; typically that limit is around 28 days. Your multidose vial may have an expiration date on the label, but that does not take into consideration the date the vial is first used, which is when the 28-day rule takes effect.
In addition, not all medications are alike—some must be used soon after being opened, which requires the manufacturer’s specific recommendations.
“The issues are that the preservatives in them may be a short-acting preservative, and so it might not be maintaining sterility for a longer period of time because if the preservation breaks down, then the bugs or the chemicals in there could be starting to take over,” Luebbert says. “So you need to worry about the shelf life based upon the sterility and stability of the agent in the vial.”
There are a few current guidelines that address expiration dates for multidose vials. The US Pharmacopeia (USP 2008), A General Chapter <797> Pharmaceutical Compounding? Sterile Preparations, requires multidose vials to be discarded 28 days after initial stopper penetration unless the manufacturer specifies otherwise. The vial should be labeled to reflect the penetration date or the beyond-use date.
However, the CDC indicates that multidose vials can be used until the expiration date, unless there are concerns with sterility.
The Association for Professionals in Infection Control (APIC) takes a stance between the two. In its document APIC Position Paper: Safe Injection, Infusion, and Medication Vial Practices in Health Care, it recommends that facilities develop policies based on one guideline or the other, then follow through with those policies.
“There are varying recommendations on when the used multidose vials should be discarded, which is why our position paper noted it as an unresolved issue,” says Susan A. Dolan, RN, MS, CIC, director of the Department of Epidemiology at Children’s Hospital in Aurora, CO, and lead author of APIC’s position paper. “Everyone would agree that it should be discarded if the sterility of the vial is in question. Prior to discarding it, though, contact your infection preventionist, as they may want the vial to culture it for microorganisms. It would be helpful to have one evidence-based recommendation for discarding used multidose vials that healthcare personnel and surveyors can both utilize.”
The regulatory perspective
On June 9, Joint Commission Online published a clarification on multidose vials, reinforcing the 28-day rule.
MM.03.01.01, element of performance 7, requires facilities to store all medications labeled with an expiration date, which is defined as the last date the product can be used, the clarification says. Therefore, facilities need to relabel the expiration date once the vial is punctured in order to reflect the 28-day rule.
CMS has also announced that inspectors will look at the 28-day window for multidose vials, and that requirement has been reflected in the “Surveyor Tool for Infection Control in Ambulatory Surgery Centers.”
Dolan notes that although APIC’s position paper straddles the line on the 28-day issue, it seems more surveyors are using that time period as a barometer.
“When we wrote our position paper, our communication indicated that staff was to follow their own hospital policy,” Dolan says. “However, that may not be the case today given conversations I had with colleagues at our recent APIC national conference. Therefore, we are currently having conversations with various organizations to help develop one consistent message for the [healthcare professional]. It appears that the 28-day recommendation will prevail along with ensuring vials are accessed, stored, and utilized properly.”
Use the smallest vial possible
Luebbert’s first recommendation is to use multidose vials as little as possible to minimize risks and complication. However, some drugs may only be provided in multidose vials by manufacturers, so there may be some cases where their use cannot be avoided. In this scenario, she recommends purchasing the smallest bott