Equip the command wand, left-click the worker (or its shrine), then click the chests you want. Click a linked chest again to unlink it, and press Escape when you're done. Stand close to each chest when you click it, and use chests you can open.
A Herdwarden takes up to 3 chests and fills them in the order you clicked. Seedsowers need a starter chest for their seeds plus at least one chest for the harvest.