Moist particles from misters and irrigation can spread legionnaires' disease (Legionellosis), and dusty soil or manure can put fungi on your lungs.
Both really hard to treat and not something you want from what I've heard.
Stuff like DE, zeolite, gypsum, rock dust, is just generally abrasive and doesn't break down readily, so its not good stuff to put in your lungs.
Minor ongoing damage for years and years. Lime is corrosive and isn't good to breath in either.
Anything that's small enough to breath in, its probably a good idea not to.
Not really a safety issue, but a word of warning.
I used to get big bags of "water crystals" from the forestry up north and use it on the garden in the sandy soil we had up there.
The dog ate about half a bag of it dry one day when I was at work.
Every time it rained for the whole time we were there great big jelly turds would spring up everywhere in the back yard.
When it stopped raining I would shovel up what I could, but it was like the bastards were breeding.
Like huge weird gross fungi, or huge jellied ants nests and the more rain we had the weirder and grosser they got.
Keep everything out of reach, as even though it was a pretty harmless jelly powder, it could have swelled in the gut and killed a kid or other dog.
Our fella was crook and sulky for ages afterwards.