The date of McCoy's birth is not known; various sources give it as March 27, 1843; May 2, 1843; and May 2, 1844. His parents, George McCoy and the former Mildred Goins, were fugitive slaves who had escaped to Canada from Kentucky.

The son of former slaves from Kentucky who had escaped via the underground railroad to Canada, at 15 years of age Elijah McCoy traveled to Scotland seeking the educational opportunities from which blacks were excluded in the Americas. He trained in mechanical engineering and then moved to the United States, where he was denied engineering employment-again because he was of African descent.

He instead took a job as a railroad fireman. At that time, locomotives needed to be shut down periodically to be lubricated to avoid overheating. The frequent stops prevented railroads from being profitable until McCoy developed the “lubricating cup” for steam engines, which kept locomotives constantly lubricated, preventing frequent stops and overheating. He patented the lubricating cup in 1872. It represented the most profitable of his more than 58 patents, which included a folding ironing board and an automatic sprinkler.

In 1920, at the age of 77, McCoy joined with investors and founded the Elijah McCoy Manufacturing Company in Detroit, serving as vice-president. The firm manufactured and sold his graphite lubricators, including an advanced version that also lubricated a railroad train's air brakes.