I recently picked up a replacement insert for my Traveler's Notebook and resisted adding a folder and a couple of extras. The setup already works, and more bulk would make it less pleasant to carry. I only needed the insert.
Keeping the setup lean also helps the inserts sit evenly under the elastic. A thicker folder can make thinner ones bow or shift; I judge additions by whether the stack still closes squarely and turns comfortably, not by insert count.
Even a few loose slips tucked into an insert can make the notebook sit unevenly. Redistributing them may fix that without removing an insert.
Tucking the loose slips into a folded sheet at the back keeps them together without adding a separate folder.