In general you are not allowed to use another person's work verbatim without permission due to copyright. In many cases you are not allowed to create derivative works either like say the New Adventures of Harry Potter.
The OGL is a type of blanket permission by the author to copy a work or portions of a work under certain conditions. The various Creative Common licenses have a similar goal.
Those conditions are:
- You must update Section 15 with the copyright notice of any open content you copy. Luckily this mostly involves copying pasting the section 15 from the license text of the work.
- You explicitly agree not to copy the work's product identify without a specific set of permission from the original author.
- You must clearly designate or mark which portions of your work is open content.
- You can't cite compatibility or use the original work for marketing without specific permission from the original author.
#4 is the only right that you are explicitly giving up by agree to use another author's open content. #4 is why Wizards had the d20 trademark license. Note: Wizards has since revoked the license and no longer gives permission to use the d20 mark or logo.
The OGL has a feature where you can designate some portions of a work as open with people free to copy that portion under the OGL, and other portions as product identity which can't be copy. This is why many opt to use the OGL over other types of license such as Creative Commons. Creative Common is designed to give permission to ENTIRE works. The OGL is deliberately designed to allow certain SECTIONS to have different permissions.
Now where it gets difficult is that people being people don't clearly mark what open content. Luckily there are documents that are marked as 100% open content that cover what most people want to do with their projects.
The Fate SRD
Swords & Wizardry SRD
The Blackmarsh Setting (which I wrote)
If you want to publish your own product. Take your rulebooks put them far away from where you do your writing.
Download one of the above (or one of the otherSRDs out there). And use that as your main reference.
Then when you lay out your book, CLEARLY mark which portions are open content. The rest is product identity in which you retain full rights.
What is open content? Any thing that is copied or based on the SRD.
For example this entry about the Boglings from my Majestic Wilderlands.
This has to be open content because it depends on information found in the Swords & Wizardry SRD and the d20 SRD.
BOGLINGSArmor Class: 6 
Hit Dice: 2+1
Attacks: 2 claws (1d3)
Saving Throw: 16
Special: Underwater, Jumping, Extensible Tongue
Move: 6/12 (when swimming)
Challenge Level/XP: 2/35
• May breathe underwater indefinitely
• Can Jump over 60 feet and up to 20 feet in height.
• Has an extensible tongue can immobilize a target if it fails it’s saving throw.
This part can be product identity because it is my own original writing.
These are amphibious humanoids with bulbous eyes. Boglings are noted for their ability to jump long distances and for their extensible tongues. They are found in tribes in tropical swamps and rainforests as well as on several of the outer planes most notably the Swamps of Acheron home to the god Set.
Personally I made the whole thing open content just because I wanted too.
When you are done, copy the text of the open game license and alter Section 15 to be
Open Game License v 1.0a Copyright 2000, Wizards of the Coast, Inc.plus whatever the section 15 the SRD has and then YOUR copyright notice. Like this from Blackmarsh
15. COPYRIGHT NOTICEA comment about Trademarks.
Open Game License v 1.0a Copyright 2000, Wizards of the Coast, Inc.
System Reference Document Copyright 2000-2003, Wizards of the Coast, Inc.; Authors Jonathan Tweet, Monte Cook, Skip Williams, Rich Baker, Andy Collins, David Noonan, Rich Redman, Bruce R. Cordell, John D. Rateliff, Thomas Reid, James Wyatt, based on original material by E. Gary Gygax and Dave Arneson.
Blackmarsh, Copyright 2011, Robert Conley
Note that if any of the following is an issue for something you are doing this the point where you need to start getting legal advice. The above is about making thing clear and easy for people to reuse other people's work without having to get lawyers involved. The below is where a lawyer comes in handy.
Trademarks are graphics and text that identify a product or business. Like copyrights they can be registered or unregistered. The White Star Logo and trademark license that the debate over on Tenkar was centered are unregistered.
Understand the law frowns on businesses pretending to be other business or trying to confuse consumers whether product is really brand X. So registered or unregistered the law protect a business right to have a distinctive mark and name for themselves and their products. What does registering a trademark get you? Namely the right to use for damages when your trademark. The best you can do with a unregistered trademark is sue to get somebody to STOP from infringing your mark.
In both cases you have to actively enforce your trademarks. The law grants powerful protection to a business' trademark but only if the business actively goes after violators. Let it slide and the law will assume you don't care and hence the violation.
This is unlike copyright where inaction can only effect your ability to recover PAST damages. Once you decide enough is enough, from that point on the clock is ticking for damages by the violator. This is of course you have registered your copyright as the best you can do with an unregistered copyright it to get them to stop distributing your stuff.
Finally do you need to deal with the OGL? No there are ways under copyright law and precedent for third parties to create compatible works. But guess what! You should go to a lawyer for advice if that the route you take. And understand if that the route you take, you will need legal advice at every step to ensure your work doesn't stray across the line and cause you legal headache. In my opinion it sounds like a huge in the pass for what is in essence a hobby pastime.
Of course some people are just dicks and rely on the fact they have little or no money or assets. Or the fact that criminal copyright has a high minimum threshold before the US Attorney will consider a prosecution. Authors dealing with such assholes have my sympathy the only thing I can offer in the way of advice is for you or your lawyer to look at the DMCA as it has provisions that help smaller publisher deal with these jerks.