Hundreds of grassroots referees have told the BBC they fear for their safety when refereeing and are dissatisfied with current measures to tackle abuse.
More than 900 referees in England responded to a Radio 5 Live questionnaire, with 293 saying they had been physically abused by spectators, players, coaches or managers.
Some described being punched, headbutted and spat at.
Almost all the respondents had experienced some form of verbal abuse.
The president of the Referees' Association in England, which distributed the questionnaire to its 7,000 members, says the abuse of match officials is having a significant impact on their mental health and they are only ever "one decision away from a smack in the mouth".
"One day in this country a referee will lose his or her life. It happened in Holland a few years ago and they really changed their culture in football," said Paul Field.
The Football Association (FA) says a small minority of people abuse referees but such behaviour is "completely unacceptable".
It has pledged to "continue to do everything we can" to stamp it out, including stronger sanctions and the launch of a three-year strategy to help tackle the issue.
Field said anyone banned at grassroots level for abusing referees should not be allowed to attend any matches and called for the FA to inform elite clubs, with a view to preventing them from buying tickets for games.
He said some parents' behaviour on the sidelines was "shocking" while players often imitate the behaviour of top-level professionals who question decisions.
There were 122 referees aged 17 and under who responded to the questionnaire - more than 100 said they had been verbally abused by coaches/managers (105), spectators (109) or players (102).
What the questionnaire revealed
Of the 927 respondents, 908 said they'd experienced verbal abuse from either spectators, players, coaches or managers.
Many (778) had been sworn at while officiating, while 375 had received personal abuse about things like their appearance, gender, race or sexual orientation.
Some 293 experienced physical abuse from either spectators, players, coaches or managers.
A threat of violence against them or their loved ones had been made against 283 of the respondents.
Verbal or physical abuse had negatively affected the mental health of 361 who answered the questionnaire.
There were 57 people who had received a death threat against them or their loved ones.
The amount of abuse towards referees is worse now than five years ago, according to 440 of the respondents.
There were 378 of the 927 referees who said they are "often" or "sometimes" worried about their safety.
Many respondents (506) were either moderately or very dissatisfied with measures currently being taken to tackle referee abuse by the English FA.
*The figures in the questionnaire are not statistically representative.
What the referees say
Ryan, 30, from Lancashire, once had to hide in a back street for an hour after being chased by players.
"Every week when you go out as a referee, you think what's going to happen this week. I sometimes don't want to turn up. It's freezing cold and you're going to stand there for 90 minutes to be abused, for £30, which is what you get paid," he said.
"Without a referee turning up at the weekend, you're not going to have football and grassroots will eventually end up going into the abyss."
Megan, 18, from Oxfordshire, had a parent come on the pitch with raised fists after she sent a child off.
"He was yelling abuse at me and saying this is why girls shouldn't be in football," she said.
"I think he got a six-match ban and fined, but that isn't enough.
"It was probably the scariest experience I've had when I've been watching, playing or reffing football. It really had an impact on my mental health, I was just worried all the time. I took two weeks off and worked my way back.
"All referees get abuse and I also get abuse because of my gender - I'm female in a male-dominated area. I've had people comment on my chest in the middle of a game."
Adrian, 59, Portsmouth: "In nearly 24 years of refereeing, I have been threatened, verbally abused, been told they know where I live, also been assaulted five times. Why I carry on I don't know - suppose the love of football."
Bill, 74, Leamington Spa: "I have been stopped from driving my car by players lying in front of my car and jumping on the bonnet. Quite a few years ago I was punched from behind and kicked on the floor. I don't look my age and have vast experience with a good reputation so I can handle myself. But younger refs are walking away."
Joe, 18, Romford: "I've been assaulted by a grown adult player when I was 16, threatened multiple times by managers and parents. A player at under-14 level threatened to have me stabbed and 'get his gang on me'. Managers have threatened to see me in the car park. Players have also said this.
"We need legislation at government level which offers us protection. The police need to take assaults on referees as criminal matters not just 'a football matter', and the FA needs to invest more and increase sanctions."
Jacob, 15, Essex: "I've had players at under-13 level swearing at me and parents undermining my authority."
'Players have a responsibility to tone it down' - Sutton
Former Premier League striker Chris Sutton - whose father was a referee - was asked by Radio 5 Live to referee his first match, a junior game, to find out what it's like on the other side.
"Any form of abuse is totally unacceptable," said the former Norwich, Blackburn, Celtic and Chelsea forward who presents the BBC's 606 football phone-in show.
"It's an issue that needs to be dealt with, otherwise grassroots football won't survive."
Sutton has admitted to previously confronting a young referee while watching his son play and now wants to address the issue.
"I walked on the field because I was concerned about how badly injured he was. I shouldn't have done that. I realise how difficult it is," he said.
Sutton said players at all levels should stop swearing at referees.
"We see Premier League and other professional players do that on a regular basis. I've done it. It doesn't make it right and all players have a responsibility to tone it down."
'You go from never being shouted at to being screamed at'
What the FA says
A FA spokesperson said: "We have over 28,000 referees in England, and they are the lifeblood of our game.
"We understand the challenges that some of them face, and we have been very clear that all forms of abuse, whether on or off the pitch, are completely unacceptable.
"While it is only a small minority of people who behave badly to referees, this is still too many, and we will continue to do everything we can to stamp out this behaviour from our game.
"Through stronger sanctions, leading innovations and a new three-year refereeing strategy coming soon, we are determined to tackle this issue and build a safer and more inclusive environment for our match officials to have happy and fulfilling long term experiences as referees."
*BBC Radio 5 Live sent the questionnaire out via the Referees' Association in England to its 7,000 members.
It was also sent out to the Referees' Association of Northern Ireland and the Referees' Association of Wales but insufficient responses were received. There is no independent referees' association for Scotland and the Scottish FA declined to send it out to its referees.
Partnerships are often formed between individuals, agencies or organisations with a shared interest to address specific issues. Our evidence reveals that our members young and old often require mental, emotional, financial, medical help, support and guidance. The Referees’ Association champion the work of those agencies that provide this support in times of need. Should you require support please contact the relevant agency below or the RA Head Office.
Join the Referees Association
The National RA recommends that prospective members should join through their local LRA, the easiest way being to attend one of the many local association meetings that take place across the country on a regular (usually monthly) basis. Find your nearest branch now.
Alternatively, online membership is now available for any referee who wishes to take up central membership. The National RA will then allocate you a local branch. Join now.