I use planbuild to run a public valheim server and most of the players are xbox players. Here is what I've learned.
You only need the mod to add the buildings to the server. After you place the buildings, other people can play on the server without the mod.
The xbox players can only join if you have the "enable crossplay" setting checked
If you try to show them something that is a mod file, like the planbuild rune, or some building plans etc, they will see nothing. The mod-specific objects will be invisible to them.