Competition for nest box

The nest box by my lawn was quickly taken by english sparrows.  My neighbor and I removed several early nests.  Then we saw a pair of bluebird take over and in fact fight off the english sparrows.  I was excited then disappointed to see the sparrows return and bluebirds disappear.  I disposed of a male sparrow but the box has remained empty.  It just goes to show how competitive the birds are for next location and the negative impact the sparrow has had on the bluebird.