Ever since latex was shown to cause allergic reaction a few decades ago, the myth has survived that sponges are either "latex-ridden" or "latex free." The issue really began with a small number of healthcare professionals that used latex products made by a dipped vulcanization process. The allergic reaction was related to the use of some medical products (like gloves) and developed into a wide-spread concern over all latex products. The hype far outweighed the harm.
Today, the cause for concern is minimal because the manufacturing process for cosmetic sponges is different. Furthermore, most factories follow strict manufacturing processes, with many subjected to thorough safety and quality audits required by their customers (i.e. major cosmetic brands).
Most high-end cosmetic sponges in the US are made of nitrile butadiene rubber (NBR). These are the sponges you typically find in compacts. Most mass market or drug store wedges are styrene butadiene rubber (SBR). These are the "bag of wedges" that are considered disposable by consumers and professional makeup artists. In neither case will you find latex allergy issues.
So does the latex allergy exist? Yes, but most commonly with non-cosmetic products. A list of products that may contain latex is provided in this link.
Want more information? Here's another helpful link about latex allergies.
So buyer beware? Always. But better yet...buyer be informed.