Radioactive decay in naturally occurring substances results in 4 types of particles flying out: electrons/positrons, photons (gamma/x-ray), alpha particles (He nuclei), and neutinos. The neutrinos escape and don't heat things up nearby--ignore them. As alphas pass through matter they ionize nearby atoms--so much so that they lose energy quickly and slow down, grab a few stray electrons and turn into a He atom. The ionized atoms recombine with nearby electrons, releasing low energy photons and generaly shaking things up a bit locally. Shaking means heat.
The electrons/positrons and photons mostly interact with other electrons around the atoms of the material they are passing through. Both typically travel much farther than an alpha, spreading their ionization along a longer track. The same sorts of things happen to the ions--they recombine, releasing lower energy photons and shaking things up a bit locally. Heat.
Sometimes the electron going through kicks the atom's electron hard enough that it in turn starts flying through the material. If the energies are high enough you can get a cascade going--and pair-produce electron/positron pairs as well. One single high energy (much higher than radioactive elements produce, btw) electron can produce a shower with thousands of particles.
Fission--let me make a little digression here. The higher the number of protons in a nucleus, the higher the fraction of neutrons you find in stable isotopes. For example, the most common iron has 26 protons and 30 neutrons: ratio is 1.15. Uranium-238 has 92 protons and 146 neutrons: ratio is 1.59 When uranium fissions, two smaller nuclei fly away from each other, with more neutrons than is good for them--and neutrons can come from the primary fission as well. The smaller nuclei try to shed neutrons, either directly or by emitting electrons (and neutrinos) to turn the neutrons into protons. So in fission you get the ordinary kinds of radiation, plus neutrons.
So, fission produces neutrons--and being neutral, neutrons don't interact much with the electrons. But they do bounce off nuclei. Bounce=random movement=heat. But in a solid, each atom has a particular place in the local lattice. If a neutron kicks its nucleus, that atom is now dislocated, and there's a gap where it originally sat. You can imagine how this effects the solid. If enough atoms get dislocated, the material tries to swell.
Neutron embrittlement can be a big problem in reactors. Designers have to choose materials carefully. I didn't see "design for replacement" anywhere, but I assume that's a factor too.
Solid structures sustain microscopic battering wherever the neutrons fly.
It had not occurred to me that other artifacts might suffer the same fate, but in Chernobyl's melted reactor, blobs of radioactive lava are disintegrating. "Early on, an FCM formation called the Elephant’s Foot was so hard scientists had to use a Kalashnikov rifle to shear off a chunk for analysis. “Now it more or less has the consistency of sand,” Saveliev says." And that results in lots of dust, too. Cleaning the place up looks very complicated.