What should I do, our social worker is biased?

I was raised by my grandparents who are abusive but never caught. Two of their children are dead (My uncle who committed suicide and my mother who overdosed.) My little brothers are in the system and we were trying to get custody of them. The grandparents called in and made unfounded reports about us. The social worker cut off my visitation with my brothers. She told me this was because of their bad behavior, but they were aloud to visit everyone but me, so I don't think it's true. Social Services did a home study on us, and now I am aloud to see them again. I'm absolutly terrified that I'll lose my visitation again. I think this social worker believes the grandparents claims. What should I do?