If you google information about Phthalates, Latex, Off Gassing and Jelly sex toys you should find lots of information.
The reason Jelly sex toys are considered not good are because they are made generally from a rubber or plastic material that has been softened using plastisizers. These chemicals do not bond well with the rubber and over time, the product can break down, or off gas, or leech. If you have a stinky jelly sex toy, that is chemicals off gassing, if you notice oily residues on your toy, or discolouration, this is leeching. Cheaper sex toys made from inexpensive crappy materials can even melt into a puddle of chemical goo over time. Storing them against other materials can 'melt' the toy.
The reason there is a big hoo ha about phthalates and chemicals in toys, is because there have been studies that suggest (but not proven) links to cancer, birth defects and damaging sperm production etc. Phthalates being used as plastic softeners were banned in certain countries from being used in childrens toys, because children put toys into their mouths. There is not the same regulations in the sex toy industry as there is in the childrens toy industry but vaginas and butts contain mucous membranes that can absorb these chemicals easily.
Anyway I am no authority on this issue, I have just done a lot of reading for a blog post I did.
Some women and men have an immediate reaction to these chemicals, suffering chemical burns to their most delicates, while others experience burning or itching and put it down to their own bodies acting up, rather than the sex toy causing irritation.
On top of this, jelly toys are generally porous, meaning they absorb chemicals, bacteria and other nasties, just in the same way it leeches chemicals. So it can never truly be cleaned and they can therefore cause infection, most especially if you share toys with another person or use the same toy to swap from anal to vaginal (Even with a wash in between)
Anyway. There is a lot of info online about this is interested. I found it interesting.
This is why I avoid jelly toys, but if you have one, I still recommend using a condom with it. Personally. If you are worried about anything.
Silicone (Pure silicone) Glass, metal, ceramic, wood = all body safe materials.