If there were an infinite amount of universe and/or an infinite universe that we live in, that means all possibilities, every single thing we can imagine, should be happening at once everywhere. That means stuff like dinosaur aliens nuking the whale empire that is hiding on earth, and a bunch of random black holes are appearing because an alien squid scientist has gone mad.
Everything, should be happening at once. Time travelers, wars, aliens, everything. Yet we live in the bubble of the oort cloud, and nothing has happened. Clearly there is no infinity out there.