Thank you for reaching out to Your Office Mom for work advice. After reviewing your question, I suspect that the project manager has expectations that you aren't meeting (through no fault of your own). Perhaps the rules of how it all was going to work were not thoroughly discussed (not unusual). Regardless, I think you need to talk to your boss since as you said your primary responsibility is your main job, not the project.
First, think long and hard to determine how much time you can realistically spend on the project each day or week. Is tracking your time part of your time management efforts? If not, consider doing that before approaching your boss. Is it feasible for you to continue to support the project? If it is, talk to your boss and explain the situation. Tell him/her you feel you can realistically spend X amount of time on the project and you want his/her approval before you talk with the project manager.
Then, talk with the PM to explain how many hours you (with your boss' approval) can realistically allocate to the project. Ask if that will be enough time to keep the tasks on track. Then, discuss how the work can be accomplished (which days or whatever) and get a clear understanding of the deadlines and determine if it is even feasible for you to stay involved based on the amount of time you have for the project. If you are still part of the project, agree on how you will keep the PM updated and how frequently.
I think this approach is logical and positive. You are not complaining; you need clarity.
As for compensation, I think it is best to talk to HR to explore your options and strategies. HR can give you insights, and if you need to approach your boss, coach you on how to do so.