Throughout the Bible, one can find references to Babylon, a Biblical city that was likely close to Baghdad -- go figure. The city was one of sin and iniquity, but Revelation also refers to Babylon repeatedly, though in a different sense.
And there followed another angel, saying, Babylon is fallen, is fallen, that great city, because she made all nations drink of the wine of the wrath of her fornication. (Revelation 14:8)
It is clear that there is some kind of symbolism here, that Babylon has thrived in some form throughout the centuries. Rastas have a unique view of so-called "Babylon." In many cases, this view manifests in the form of the government. Rastas have faced persecution since Rastafari was formed in the early part of the century. This persecution comes in many forms, such as ganja possession. It is common knowledge that Rastas use ganja, and therefore have become targetted for their use.
Also, as Rastas are generally very attached to black pride and their African roots, they often fall under the statistic of being oppressed because they are black (or many are). There is a lot of merit to this, and it is true that Rastas are often persecuted by police. That's Babylon. Also, the slavery imposed by whites that brought the righteous Africans from their blessed continent is Babylon. The term Babylon is often used to refer to some kind of oppression, or ill-treatment from Western society.
Anything that is unholy is Babylon. It is the evil that will be wiped out when all of Revelations comes to pass. In some Rastas' eyes, this includes the government, homosexuals, technology, even members of other religions. It depends on the Rasta, however, and this traditional perspective comes mainly from old-time Jamaican Rastas.