So, If Money is Speech . . .

Our prevailing legal framework holds that money is speech. This means that since money is needed to get a message out there, for example in political campaigns, the restriction of money is a restriction of speech. This suggests that access is indivisible from speech itself. Wouldn't it follow, then, that people without money are having their right to free speech infringed upon? I'm not trying to be cute here. It seems like this makes legal sense, no?