I think you should go for it! You may have some obstacles along the way but I think from the sounds of how you feel you know in your heart it is the right thing to do.
I have a neighbor who has 4 children of his own and last year took custody of his sister's 3 children; she was on drugs and in and out of trouble. She has caused some trouble but not much really and the kids are in a solid home with lots of love and are doing well after the hell they have been through.
Hang in there, pray about it, let God show you the way.