Brittleness is one of the problems that limits usage of concrete in some cases. When ordinary concrete reaches to peak load and cracks, it disintegrates and does not sustain load anymore. Researchers have added different materials to the concrete to overcome this problem. Adding fibers is one of the ways to increase its energy absorption capability. Because of high ductility and deformability of rubber and polystyrene, these materials have been used to increase ductility of concrete. Different amounts of fine aggregates in ordinary concrete were replaced with two sizes of rubber powders and two sizes of polystyrene. Also different amounts of coarse aggregates were replaced with shredded rubber in three sizes, to investigate the effect of quantity and size of these aggregates on the energy absorption capability of concrete. In addition to assessing ductility, workability, unit weight, compressive strength, and flexural strength were evaluated. For this purpose, 336 cubic specimens and 168 prism specimens were molded and tested. Flexural toughness tests were done to study energy absorption capability according to ASTM and Japan standards. Experimental results showed that when the concrete containing rubber and polystyrene reaches to the peak load and cracking happens, it sustains lower loads and does not fully disintegrate. Also the results showed that in the concrete containing shredded rubbers, toughness factor I 50 was increased up to 35 and residual strength factor increased up to 70%.