A survey by UCU found that 42% of academics, and 76% of non-UK EU academics, said they are now more likely to consider leaving UK higher education.
Almost one third (29%) of respondents said they know of academics who are already leaving the UK, and 44% said they know of academics who have lost access to research funding as a direct result of the Brexit vote.
90% of respondents believe Brexit will have a negative impact on the UK higher education sector, UCU said.
Sally Hunt, general secretary of UCU, said: "I am deeply worried that so many academics already know of staff leaving as a result of the Brexit vote, and that three-quarters of EU nationals are now considering leaving the UK."
The government must "try to retain the talented academics working in this country by guaranteeing EU staff already working in the UK the right to remain", UCU said.
The UCU survey also asked respondents about the Higher Education and Research Bill that is due to be debated in the House of Lords.
The bill had its first reading in the House of Commons in May. If passed in its current form, the bill would enact reforms including simplifying the processes through which new institutions may be given degree-awarding powers and university title, and providing prospective students with more information about teaching standards and job prospects up front
Three-quarters of respondents (76%) said the bill's plan to link the Teaching Excellence Framework to tuition fees would have a negative impact on higher education.
Almost two-thirds of respondents said student satisfaction is an ineffective or very ineffective measure of teaching quality, 55% said graduate employment is also an ineffective or very ineffective measure and 59% did not believe student dropout rates would be an effective measure of the quality of teaching, UCU said.
The union called on the government "to require new providers to demonstrate a track record of higher education delivery before gaining degree-awarding powers", it said.
Higher education expert Martin Priestley of Pinsent Masons, the law firm behind Out-Law.com said: "Retaining free movement of academic staff and access to EU research funds are issues of critical importance to the sector. Universities are concerned not just with retaining existing staff but also with attracting talented academics from the EU. There is fierce competition for academics and anecdotal evidence suggests that UK universities are struggling to retain and recruit the best. Brexit is causing academics from outside the UK who previously would have accepted offers to join UK universities to rethink their options."