Good manners do not contemplate a hostess’s inviting people to only part of an event. Inviting people to one event and not to another is actually fine (despite what you often read on wedding boards), but once guests are at an event, how long they choose to stay is up to them*. So for your plan to succeed, you need to influence those choices.
One way to influence your siblings’ choices would indeed be word-of-mouth. Since bluntly ordering your second-class guests to leave early (or even admitting that you consider them to be a second class of guests at all) would be rude, you will need to approach the subject tactfully, recognizing your own family dynamic, which naturally you know better than we do. In some families, expressing to your siblings your worry that as the night goes on the children might be exposed to too much boisterous partying, and that they might cramp their parents style, might lead the parents to offer to have the children taken home. In other families parents might just blow off such worries, and be unoffended by something more direct. You will have to judge.
Another way to influence the parents’ choice, would be to address the logistical complexities that pixiecat mentions. Letting parents know, for example, that you will be providing bonded child-safe taxi service to take children home at 8:30, might inspire the parents to send them at that time.
Or offering up a hotel-room or mini-suite at the venue with a certified nanny watching over it, to open at 8:30 where parents can take their children to watch family-rated movies, eat popcorn and, for the younger ones, go to sleep; will let you strongly hint to your siblings that since the suite is provided, they should have their children use it. This will work particularly well, if you poll your nieces and nephews to find out what their favourite movies are (particularly the ones that their parents will tolerate but haven’t actually permitted at home yet) and announce to the children at the dance itself that the “kid’s movie suite” is now open and “Nanny Susan” is ready to lead them all to the suite. The parents would probably have to fignt to hold their kiddies back at that point, and you win.
Or, you could offer a seat at dinner to their various babysitters on the expectation that the babysitters will drive the children home and watch the children at their own homes. You can broach that with your siblings by asking what time fits best with the childrens’ bedtimes: eight o’clkock or eight-thirty? If you decide to propose one of these options, that too should be done in conversation rather than in a note to allow you tactfully to adjust your words when you sense how siblings are reacting to the proposal.
A third option, if your ceremony and wedding dinner-dance are at two different locations, is to follow the traditional pattern of a single daytime ceremony-and-reception event at the same location, to which the children are invited; and a separate evening dinner-dance event where you and your adult guests get snockered. You would serve tea or punch and cake at the reception, and toss your boouquet as you leave the ceremony site, thus giving the children everything that they know about weddings: the pretty dresses, the towering tiered cake with the bride and groom cutting the first slice, and the scramble for the flowers. Tieredwedding cakes work particularly well for this traditional format, since the largest tier can be served at the reception which has the largest number of guests, and the remaining cake still looks complete and elegant when set up at the wedding-breakfast or dinner-dance location.
*subject to the limits that they must not stay after the party has wound down and the hosts are yawning and restless to say goodnight, and they must not leave before the guests of honour leave. Since most modern weddings do not have guests of honour, this latter is irrelevant.