Preheat your oven to 350°F (175°C). Grease and flour a 9-inch round cake pan.
In a medium bowl, whisk together the flour, baking powder, baking soda, and salt. Set aside.
In a large mixing bowl, cream the softened butter and granulated sugar together until light and fluffy, about 3-4 minutes.
Add the eggs one at a time, mixing well after each addition. Stir in the vanilla extract.
Gradually add the dry ingredients to the wet mixture, alternating with the buttermilk. Begin and end with the dry ingredients. Mix until just combined.
Gently fold in the diced peaches and raspberries, being careful not to break the raspberries.
Pour the batter into the prepared cake pan and smooth the top with a spatula.
Bake in the preheated oven for 25-30 minutes, or until a toothpick inserted into the center comes out clean.
Allow the cake to cool in the pan for 10 minutes before transferring it to a wire rack to cool completely.
Once cooled, dust with powdered sugar if desired, slice, and serve.