Heat 2 tablespoons of olive oil in a large Dutch oven or pot over medium heat. Add the fennel and the onions, and sauté for 5 minutes until they start to become tender. Add the zucchini and the garlic, season with salt and pepper, and sauté for another 5 minutes, until all of the vegetables are tender. Add the white wine, and stir to scrap up any browned bits stuck to the bottom of the pan.
Add the tomatoes with their juice, broth, thyme and saffron bring to a simmer; add the potatoes and simmer for 10 to 12 minutes, until the potatoes are tender.
Meanwhile, heat the remaining 2 tablespoons olive oil in a large skillet over medium heat. Season the fish with salt and pepper, and sear the filet on both sides, for 5 to 7 minutes per side, until the fish is just barely cooked through (it really depends on the thickness of the fish - if it falls apart a little that's a-ok). Remove and set aside.
When cooled slightly, use a fork to break the fish into 1-inch chunks. Add them to the pot along with the orange zest and juice, and halved olives, stirring gently to combine.
Serve hot with slivered basil on top of each portion, ladled into individual bowls. Spoon some rouille on the slices of barguette and place them on the side or on top of the bowls.