I think what I am doing is fair use, but what if someone says it isn't?

There is some protection for unintentional infringement by educators in a not-for-profit use. Keep careful records. What part of the copyright law or guidelines led you to believe it was fair use? What steps did you take to locate the copyright owner? What attempts did you take to secure an original copy of the item? In other words, document your good faith effort to stay within the copyright law and guidelines.