Did you buy something that wasn't for you? Returning that item might get a lot harder soon, as retailers are cracking down on their No. 1 concern.

"Fraud is No. 1, and it's not even close to No. 2," Pitney Bowes Vice President Vijay Ramachandran told CNN.

A survey by the Appriss Retail and the National Retail Federation found that retailers estimated about 13.7% of returns were fraudulent last year, spiking even higher during the peak of the holiday season to 16.5% of returns, wiping $24.5 billion off retailer profits during their most profitable time of the year.

The most common form of return fraud, accountable for about half the fraud, is known as "wardrobing," when consumers return nondefective items after temporary use. For example, buying a heater in the winter months only to return it when the weather warms.

This type of fraud might hit especially close to home. According to Forter, a company specializing in fraud prevention, 56% of consumers confess to this act.

Accounting for a third of return fraud is the return of stolen items and then receiving cash or store credit in return.

Of course, return fraud isn't the only type of fraud retailers have called out in recent years.

Walmart Inc. (WMT  ) CEO Doug McMillon warned in December 2022 that "prices will be higher and/or stores will close" in the face of rising fraud.

To combat return fraud, companies often need more sophisticated methods than just locking items behind glass or hiring increased security in the store.

How far companies go in their war on combating return fraud will need to strike a balance between protecting profits and not alienating their honest customers.

"It depends on economic sentiment, consumer confidence, whether or not the retailer is financially well-positioned or not, how much inventory they have in stock, etcetera," Sucharita Kodali, a VP analyst at the research and advisory company Forrester, told NPR in December.

Given that 73% of shoppers pick a retailer based on their return experience, according to a survey by Appriss Retail and Incisiv, deep-pocketed companies making strategic investments to combat actual fraud without punishing everyone might be best poised to capitalize.

Take Amazon.com Inc. Amazon's (AMZN  ) vice president in charge of seller services recently stated the company acts by "issuing warnings, closing accounts and preventing individuals who engaged in refund fraud from opening new accounts." To help do so, it spent $1.2 billion in 2022 and employed 15,000 people to fight theft and abuse, all while using machine learning to better detect and prevent fraud.

Others are straight-up increasing the cost of making a return.

At Kohl's Corp. (KSS  ), customers whose merchandise was purchased on Kohls.com do not get their shipping costs refunded, and only items purchased on the website can be returned by mail - if bought from a store, returns need to be done at a store.

Adding friction to the returns process might frustrate honest customers, especially having to navigate through differing return policies at different retailers, but it has become more necessary as both retailer costs and overall fraud targeting retailers have increased in recent years.