The origin of breast cancer, whether primary or recurrent, is unknown. Here, we show that invasive breast cancer cells exposed to hypoxia release small extracellular vesicles (sEVs) that disrupt the differentiation of normal mammary epithelia, expand stem and luminal progenitor cells, and induce atypical ductal hyperplasia and intraepithelial neoplasia. This was accompanied by systemic immunosuppression with increased myeloid cell release of the alarmin S100A9 and oncogenic traits of epithelial-mesenchymal transition, angiogenesis, and local and disseminated luminal cell invasion in vivo. In the presence of a mammary gland driver oncogene (MMTV-PyMT), hypoxic sEVs accelerated bilateral breast cancer onset and progression. Mechanistically, genetic or pharmacologic targeting of hypoxia-inducible factor-1α (HIF1α) packaged in hypoxic sEVs or homozygous deletion of S100A9 normalized mammary gland differentiation, restored T cell function, and prevented atypical hyperplasia. The transcriptome of sEV-induced mammary gland lesions resembled luminal breast cancer, and detection of HIF1α in plasma circulating sEVs from luminal breast cancer patients correlated with disease recurrence. Therefore, sEV-HIF1α signaling drives both local and systemic mechanisms of mammary gland transformation at high risk for evolution to multifocal breast cancer. This pathway may provide a readily accessible biomarker of luminal breast cancer progression.