Bring a large pot of salted water to a boil. Add the potatoes and cook for 7 or 8 minutes, until they are just barely tender. Drain and cool.
Meanwhile, place the oil, shallot, yogurt, dill, jalapeno, lemon juice, garlic, salt and pepper in a food processor or blender and process until smooth.
In a large serving bowl combine the arugula, potatoes, cucumbers, radishes and olives. Drizzle over the dressing and toss to combine. Top with the chopped fresh dill.
Notes
If you wanted to make the dressing up to 4 days ahead of time and keep it in the fridge you definitely could. You could also cook the potatoes and slice the cucumbers a day ahead, and then just toss the whole thing together before serving. This would be excellent on a picnic or as a potluck offering.