Understanding the cellular and genetic mechanisms driving human-specific features of cortical development remains a challenge. We generated a cell-type resolved atlas of transcriptome and chromatin accessibility in the developing macaque and mouse prefrontal cortex (PFC). Comparing with published human data, our findings demonstrate that although the cortex cellular composition is overall conserved across species, progenitor cells show significant evolutionary divergence in cellular properties. Specifically, human neural progenitors exhibit extensive transcriptional rewiring in growth factor and extracellular matrix (ECM) pathways. Expression of the human-specific progenitor marker ITGA2 in the fetal mouse cortex increases the progenitor proliferation and the proportion of upper-layer neurons. These transcriptional divergences are primarily driven by altered activity in the distal regulatory elements. The chromatin regions with human-gained accessibility are enriched with human-specific sequence changes and polymorphisms linked to intelligence and neuropsychiatric disorders. Our results identify evolutionary changes in neural progenitors and putative gene regulatory mechanisms shaping primate brain evolution.