Huddersfield Rugby League Referees Society said it no longer felt that "league discipline will safeguard" its under-18 officials.
Junior ref Dylan, 14, said he was sworn at by a parent during a children's game last month, leaving him "upset".
Yorkshire Junior and Youth League said it was working "to improve discipline".
However, referee societies in West Yorkshire say officials, many who are children themselves, are giving up the sport because of abuse.
Huddersfield Rugby League Referees Society said it had tried working with the league to keep rugby enjoyable for everyone but "no progress has been made".
Meanwhile, Jayden Covell-Wood, from Dewsbury and Batley Referee Society, said: "We don't feel safe sending our younger referees to a game when it's happening more regularly.
"It's getting too much. We're losing referees because of it. Some last two or three games before they're abused and it's just not good enough."
Mr Covell-Wood, who is a rugby league referee himself, said the abuse had come from parents and coaches during under-10 games.
"I'm absolutely appalled at times. I just don't know what goes through an adult's head to even speak to a child that way," he said.
Junior referee Dylan took part in an officials' course earlier this year, but despite only being involved in a handful of games he has already been subjected to abuse.
"The coaches and parents like to moan at every single call you give," he said. "Sometimes I'll get abuse and I'll just want to end the game right there and go home."
His father Kevin said he noticed the abuse was getting worse, with his son being shouted and sworn at by a parent during a recent game.
He said: "There's a lot of under-18 refs who are only lasting a few matches.
"If they're facing this sort of abuse in the first few games, it's not good for the sport. Without the referees you don't have a game."
He said some parents felt because the young referees were paid they were open to be shouted at.
"You think 'why am I bothering to give my time up?' What makes a grown man or woman think they can verbally abuse a minor?"
Yorkshire Junior and Youth League said it was working with the Rugby Football League and clubs to improve discipline across the country.
It acknowledged behaviour on the touchline "is not where it needs to be" and there was "more work to be done".
"In 2021 we have introduced a number of additional measures, including suspension of fixtures and more education for coaches," a spokesperson said.
Harsher sentences for clubs who abuse referees, including being kicked out of competitions, were options open to the league, it added.
Our Partnerships
Partnerships are often formed between individuals, agencies or organisations with a shared interest to address specific issues. Our evidence reveals that our members young and old often require mental, emotional, financial, medical help, support and guidance. The Referees’ Association champion the work of those agencies that provide this support in times of need. Should you require support please contact the relevant agency below or the RA Head Office.
Learn more
Join the Referees Association
The National RA recommends that prospective members should join through their local LRA, the easiest way being to attend one of the many local association meetings that take place across the country on a regular (usually monthly) basis. Find your nearest branch now.
Alternatively, online membership is now available for any referee who wishes to take up central membership. The National RA will then allocate you a local branch. Join now.