Generally, this is true. It's not always true. We need to account for that. If I recall the theory correctly, it's that the Y chromosome has a big part in triggering the change from female to male. It is always possible to have a defective Y, or it's also possible for this trigger to be external. You do get the occasional girl with XY or boy with XX. It's rare, but it does happen. It's not necessarily fair to say that we all start out as female, but without this trigger we all would become female. It's just a LITTLE more complex than simply XY is male and XX is female. That's a good way to think about the basics, but the actual science runs deeper.
We can think of it a lot like Newton's Laws. They are great for making calculations, but they will eventually fall apart in some circumstances (say, as we approach the speed of light), so we use Einstein's Laws to FURTHER understand things. But even Einstein's Laws will break down (say, inside a black ****).