You can choose not to found a Religion simply by not researching the tech that unlocks it. If you need that tech but do not want the Religion, you're out of luck. If, for some strategic reason, you want another civ to found that Religion instead of you, you must wait until they research the tech.
You can take measures to ensure that one of "your" Religions never has any direct impact on the game. Religion does not spread by wildfire, you have to coax it along with Missionaries. Furthermore, a certain Civic prevents non-State Religion spread. Finally, you do not necessarily need to build the Shrine that corresponds to a Religion you've founded.
Economically, it's advantageous to be the founder of a Religion. Diplomatically it's your State Religion that's important. So overall I would say that the negative consequences of founding a Religion are minimal (can you figure out what these are?!).