Heat 2 tablespoons of olive oil in a large Dutch oven or pot over medium heat. Add the fennel and the onions, and sauté for 5 minutes until they start to become tender. Add the zucchini and the garlic, season with salt and pepper, and sauté for another 5 minutes, until all of the vegetables are tender. Add the white wine and stir to scrape up any browned bits stuck to the bottom of the pan.
Add the tomatoes with their juice, broth, thyme, and saffron. Bring to a simmer; add the potatoes and simmer for 10 to 12 minutes, until the potatoes are tender.
Meanwhile, heat the remaining 2 tablespoons olive oil in a large skillet over medium heat. Season the fish with salt and pepper, and sear the filet on both sides, for 5 to 7 minutes per side, until the fish is just barely cooked through (it really depends on the thickness of the fish — if it falls apart a little that's a-ok). Remove and set aside.
When cooled slightly, use a fork to break the fish into 1-inch chunks. Add them to the pot along with the orange zest and juice and halved olives, stirring gently to combine.
Serve hot with slivered basil on top of each portion, ladled into individual bowls. Spoon some rouille on the slices of baguette and place them on the side or on top of the bowls.
Notes
The saffron is optional mostly because of its cost. Saffron is a wonderful but pricy spice – it’s made from the stamens of crocuses, which might not sound delicious, but it adds a flavor that is at once elusive, rich, aromatic, and deep. And even though it’s expensive, you will see that a little goes a long way. Just a pinch of saffron adds so much dimension to the soup and any fish or seafood soup or stew.
You can make the soup base up to three days ahead of time, through Step 2. Stor covered in the ridge. Leftovers can be kept in the fridge for up to 2 days. Reheat on the stove over low heat just until warmed through.