Canning Town, London born inside left James Dillimore began his football career with Barking Town, and played for Brentford in 1917 as a wartime guest during the First World War, scoring 5 goals in 13 appearances, but only got an opportunity in League football when signed by Third Division (South) club Millwall in 1922, for whom he made his Football League debut against Charlton Athletic the same November, scoring in a 1-1 draw. Playing at centre forward he scored in each of his first three matches for The Lions, but soon switched to inside left on a regular basis from the start of the 1923-24 campaign. During that season he scored a career best 16 goals including a hat-trick against Brentford in November 1923, second top scoring behind Harry Morris.
He scored double figure goal return totals in each of the next two seasons but in January 1926, having scored a hat-trick in a win over Southend United in September 1925, he suffered a career ending injury and was forced to retire from League football after 42 goals in 104 appearances for Millwall. He was however fit enough to join non league club Weymouth later the same year and played for them until his eventual retirement.