Conference Fees for Speakers

Listening to a podcast on open access I had an idea: Many conferences waive conference fees (which can be substantial) for invited speakers. But those are often enough the most senior people who would have the least difficulty in paying the fee from their budget or grant money. So wouldn't it be a good idea for conferences to offer to their invited speakers to instead waive the fee for a graduate student or junior post-doc of the speakers choice and make the speaker pay the fee from their grant (or reduce the fee by 50% for both)?

Discuss!