Tag and rag was a relatively common expression in the 16th and 17th centuries, and it was often used pejoratively to refer to members of the lower classes of society. By the 18th century, the phrase