Thaw fish, if frozen. Rinse fish; pat dry with paper towels. Set aside. In a small saucepan cook the green onions and garlic in 2 tablespoons hot oil for 1 to 2 minutes or until onions are soft and garlic is fragrant. Remove saucepan from heat. Crumble saffron threads into oil mixture; stir. Remove from heat and let stand to infuse the saffron.

Meanwhile, cut a thin slice from one end of each orange and the lemon so fruit will sit level. Working on a cutting board, cut down from the top of the fruit to remove peel and white part of the rind. Working over a bowl to catch juices, remove the sections by cutting into the center of the fruit between one section and the membrane; cut along the other side of each section next to the membrane to free the section. Remove seeds.

Add saffron oil to bowl with fruit sections and juices. Stir in mint, 1/4 teaspoon salt and 1/4 teaspoon pepper.

Lightly brush both sides of swordfish steaks with additional olive oil; sprinkle with the remaining 1/4 teaspoon salt and the remaining 1/4 teaspoon pepper.

For a charcoal or gas grill, grill fish on the rack of a covered grill directly over medium heat for 8 to 12 minutes or just until fish flakes when tested with a fork, turning once halfway through cooking time.

Gently stir the fruit-mint mixture to combine; spoon over fish. Serve immediately.