If the bishop ignores my complaint that a priest broke the seal of confession, does it just come down to his word against mine?

The bishop does have an obligation to explain what he is doing, at least in a formal letter telling you that he does not judge your complaint to register as a breaking of the seal, though he may not go into why, because that can open messy matters. But what he should do and what he does do are two different things: many bishops are political animals who will not overextend themselves, and a genuinely honest bishop who follows canon law to the letter is rare today. If a decision has to be made about whether the seal was broken, there are contingencies and doubts involved, and he is the one to judge them; you have to accept his decision unless you appeal to Rome, to the Apostolic Signatura, which only takes very serious cases backed by hard evidence, not mere allegations. The bishop is also right to require solid proof, since people lie and may simply want a disliked priest removed, and with priests already in short supply he must weigh losing another one. So there is not much further you can go, but if you are truly worried, do not go back to that priest for confession.