Using a blender, process garlic, bread, cayenne pepper, saffron, and egg yolk to a paste. Work in 2 tablespoons (30ml) liquid from fish broth pot to thin paste slightly. (This can also help a blender or food processor work the mixture more effectively.) Season with salt.
Transfer While processor running at its lowest speed, drizzle in the olive oil. If the mixture seems too thick at any point, add fish broth, 1 tablespoon (15ml) at a time, until a thick, mayonnaise-like consistency is reached. Refrigerate the rouille until ready to use, up to 3 days.