The explicit description of homogeneous operators in the Cowen-Douglas class and the localization of a Cowen-Douglas Hilbert module naturally leads to the definition of a class of Cowen-Douglas operators possessing a flag structure. These operators are irreducible. We show that the flag structure is rigid in the sense that the unitary equivalence class of the operator and the flag structure determine each other. We obtain a complete set of unitary invariants which are somewhat more tractable than those of an arbitrary operator in the Cowen-Douglas class.