Up to 40 countries could boycott the next Olympic Games, making the whole event pointless, said Poland's sport and tourism minister Kamil Bortniczuk.
His comments came after Poland, Lithuania, Estonia and Latvia jointly rejected an International Olympic Committee (IOC) plan to allow Russians and Belarusians to compete in 2024.
Ukraine has threatened to boycott the Paris Olympics if that occurs.
But the IOC said on Thursday that any boycott would only "punish athletes".
Bortniczuk said he believed it would be possible to build a coalition of 40 countries, including Great Britain, the United States and Canada, to support a block on the IOC's plans before a meeting on 10 February.
He added: "Considering this I don't think we will face tough decisions before the Olympics and, if we were to boycott the Games, the coalition we will be a part of will be broad enough to make holding the Games pointless."
The IOC announced last week that it would "explore a pathway" to allow Russian and Belarusian athletes to compete in Paris under a neutral flag, adding that "no athlete should be prevented from competing just because of their passport".
The move prompted condemnation, with the UK Government saying the plan was a "world away from the reality of war".
Ukraine sports minister Vadym Guttsait said the country's sporting bodies needed to "strengthen communication" with international federations to keep a ban in place on Russian and Belarusian athletes, which was imposed by the IOC's executive committee immediately after Russia's invasion of Ukraine in February 2022.
However, IOC president Thomas Bach has since said that was only intended as a "protective" measure towards those athletes, and now insists they should not be discriminated against.
On Thursday, sports ministers from Latvia, Lithuania, Estonia and Poland said "any effort by the International Olympic Committee to bring back Russian and Belarusian athletes to compete, even under a neutral flag, should be rejected".
They added: "Efforts to return Russian and Belarusian athletes to international sports competitions under the veil of neutrality legitimize political decisions and widespread propaganda of these countries."
And they called on "all international sports organisations and federations" to remove Russian and Belarusians athletes from international competitions until the war ends.
The United States government said it supports suspending Russian and Belarusian sport governing bodies from international sports organisations and is also encouraging organisations to stop broadcasting events into both countries.
White House press secretary Karine Jean-Pierre added that, if athletes are permitted to participate in events such as the Olympics, it should be as neutral athletes and "it should be absolutely clear that they are not representing the Russian or Belarusian states".
The IOC reiterated that no discussions on the return of Russian and Belarusian athletes to competition had yet taken place and warned Ukraine and other nations about the implications of threatening a boycott.
"Threatening a boycott of the Olympic Games, which the NOC of Ukraine is currently considering, goes against the fundamentals of the Olympic movement and the principles it stands for," the IOC wrote in a question-and-answer document published on Thursday.
"A boycott is a violation of the Olympic charter, which obliges all NOCs to 'participate in the Games of the Olympiad by sending athletes'. As history has shown, previous boycotts did not achieve their political ends and served only to punish the athletes of the boycotting NOCs."
BDST: 1721 HRS, FAB 03, 2023