A new condition imposed by the European Commission this month is likely to prove impossible to meet, the LSE has said.
The Commission said that the LSE must commit to selling its majority stake in electronic trading platform MTS to gain clearance for the merger.
The LSE "tested thoroughly the feasibility and implications" of the "disproportionate" request, and concluded that it would not be able to meet the deadline for putting together a proposal on the sale.
MTS is a "systemically important regulated business in Italy due to its significant role in the trading of Italian government bonds and other securities," the LSE said.
"Any change of control of MTS would require, in particular, the approval of the Italian authorities and would trigger parallel regulatory approval processes in other jurisdictions including the UK, Belgium, France and the USA," it said.
After discussions with the Italian authorities, the LSE board believes it is unlikely that a sale could be achieved satisfactorily, "even if LSE were to give the commitment", it said.
"Moreover, the LSE board believes the offer of such a remedy would jeopardise LSE's critically important relationships with these regulators and be detrimental to LSEs ongoing businesses in Italy and the combined group, were the merger to complete," it said.
The LSE acknowledged that the Commission was now unlikely to agree to the merger.
By combining the exchanges of Germany, the UK and Italy, as well as several of the largest European clearing houses, it would create by far the largest European exchange operator, the Commission said in September.
The European Commission is concerned that the proposed merger would reduce competition in several financial market infrastructure areas.
Competition expert Guy Lougher of Pinsent Masons, the law firm behind Out-Law.com said: "Competition assessments are driven by the overlaps identified between the merging parties. The merger will not be approved unless the parties agree to remedy fully the competition issue identified by the Commission, and divesting the problematic business is the obvious remedy."
"Analysis of this kind of deal is very specific to each particular case. The reality is the precedent impact of such a prohibition on other M&A deals is limited, simply because the competition analysis in each merger is so fact-dependent," he said.
Deutsche Börse is best known for operating the Frankfurt Stock Exchange but also runs other regulated exchanges, most notably the Eurex and EEX exchanges, trading various types of derivative products. Apart from trading its activities include the supply of post-trade infrastructure services such as clearing, settlement and custody services.
LSE operates the London Stock Exchange but also owns Borsa Italiana, the Italian stock exchange, and operates a number of other trading platforms for trading of stocks, other equity-like exchange traded products, bonds and derivatives. LSE is also active in the post-trading space, most notably in clearing through the London Clearing House and through Cassa di Compensazione e Garanzia, the Italian clearinghouse.
The exchange operators agreed on the proposed £21 billion merger in February 2016, and recommitted to the deal the day after the UK referendum on 23 June. Shareholders of the LSE then "overwhelmingly" voted to approve the merger on 5 July.
Shareholders in Deutsche Börse gave approval for the planned merger in July 2016, after Deutsche Börse lowered the minimum acceptance threshold to make sure that the vote succeeded.