Rhizomes are essential organs for growth and expansion of Phragmites australis. They function as an important source of organic matter and as a nutrient source, especially in the artificial land-water transitional zones (ALWTZs) of shallow lakes. In this study, decomposition experiments on 1- to 6-year-old P. australis rhizomes were conducted in the ALWTZ of Lake Baiyangdian to evaluate the contribution of the rhizomes to organic matter accumulation and nutrient release. Mass loss and changes in nutrient content were measured after 3, 7, 15, 30, 60, 90, 120, and 180 days. The decomposition process was modeled with a composite exponential model. The Pearson correlation analysis was used to analyze the relationships between mass loss and litter quality factors. A multiple stepwise regression model was utilized to determine the dominant factors that affect mass loss. Results showed that the decomposition rates in water were significantly higher than those in soil for 1- to 6-year-old rhizomes. However, the sequence of decomposition rates was identical in both water and soil. Significant relationships between mass loss and litter quality factors were observed at a later stage, and P-related factors proved to have a more significant impact than N-related factors on mass loss. According to multiple stepwise models, the C/P ratio was found to be the dominant factor affecting the mass loss in water, and the C/N and C/P ratios were the main factors affecting the mass loss in soil. The combined effects of harvesting, ditch broadening, and control of water depth should be considered for lake administrators.