Why does it feel rude to refuse tea but you don't want any?
The short answer
Tea is a hospitality protocol. Accepting acknowledges the social relationship. Refusing signals rejection of the person, not the drink.
Tea in Morocco is a social contract. Offering tea means you're a guest. Accepting it means you recognise the relationship. The tea creates a space — ten minutes, three glasses — where two people sit as equals. Business may happen. It may not.
Leaving after the first glass causes no offence. Leaving before the first glass is finished is rude. Staying for all three and buying nothing is entirely acceptable. Tea is a cost of doing business, not an invoice.