Perhaps it's the liver, honey and almonds ragout promised to guests and hurled to the floor, or the Groundhog Day repeated arrival of those guests, or the fact that the frightened staff have bolted, but the warning signs that this glamorous dinner party is far from normal are loud and clear.

Such alarms start early: the bell that welcomes ticket-holders to The Exterminating Angel at the Royal Opera House proves to be sounding in the auditorium too, and it rings right into the opening chords of the UK premiere of Thomas Adès's new opera. From charm to chaos and worse is a repeated refrain in this compelling and dystopic work: the chunky real-live sheep on stage can only come to a sticky end. The besotted engaged couple too. Even designer Hildegard Bechtler's gorgeous, covetable gowns are in peril.

We join the rich, powerful or achieving at the party thrown after a performance of a Donizetti opera – the nobility, an explorer, a physician ... Most of the guests are unknown to each other, but they have all seen the show, and its star is the guest of honour – Leticia, sung by American coloratura soprano Andrea Luna. How in an opera do you mark out the character who is an opera singer? – Have her sing an octave higher than the rest, an accomplishment that the aptly-name Luna pulls off skilfully and often hilariously, although her final and important aria would have been more powerful sung at a 'normal' pitch.

Larky, flirty hostess Lucia (Amanda Echalaz) has practical jokes lined up for her guests, but the gradual discovery that none can leave the room outdoes any such playfulness. Polite society rapidly descends into a pit of mutual contempt, self-preservation, crude improvisation, suspicion and improvisation. Throw another cello on the fire!

Packed with cultural references, Adès, who conducts the Orchestra and Chorus of the Royal Opera in his own opera, and his librettist and director Tom Cairns, love a joke at the expense of their own profession: 'What's one conductor more or less in the world?' asks a character, dismissively.

Bechtler's visual feast – the 1960s in a nutshell – and Jon Clark's ghostly lighting point up the contrast between the superficial and the supernatural. A Rothko-like feathery block of white slides across the stage between scenes, a flying figure swoops like something from a Chagall, and creeping, detached fingers remind us of the opera's roots: the 1962 film El Angel Exterminador by the famed surrealist Luis Buñuel, who loves the image of a crab-like hand going about its own sinister business.

(Buñuel wanted to film William Golding's story of a society in freefall, Lord of the Flies, which this opera so much resembles, but failed to obtain the rights, and made El Angel Exterminador instead. Peter Brook clinched the Golding a year later.)

Adès's music, too, is stuffed with allusions, none so merry as the riff on JS Bach's Sheep May Safely Graze (oh no they mayn't!). Strident processional brass and drums, the ethereal ondes martenot with its music of the spheres, and a Spanish guitar augment an already huge orchestra, spilling up into the side boxes above the pit. Here is a smattering of Richard Strauss, of Bernstein, of Wagner. And there are moments of great beauty in this mayhem, notably the macabre love duet soprano Sophie Bevan and tenor Ed Lyon, who find their own way out.

It is never made clear what stops the characters leaving the room, but there are many allegorical possibilities. The inability of the privileged to function without the servitude of others is certainly an early contender, and the idiocy of superstition only makes everything worse. Art, it seems, finally unlocks the door. But above all it is the disabling nature of intolerance of others that threads through this grim fantasy and makes it horribly and powerfully relevant.

In a huge cast of largely home-grown soloists, there are outstanding performances from John Tomlinson as the ponderous doctor, counter-tenor Iestyn Davies and soprano Sally Matthews as incestuous siblings, US tenor Charles Workman as the host Nobile, and Canadian tenor Frédéric Antoun as the explorer.

Sung in English, with English surtitles, this impressive example of co-operation between major international opera house is a co-production with Salzburg Festival, the Metropolitan Opera, New York, and the Royal Danish Opera. It's a really big event in the British musical calendar with a great deal to say about living in harmony – or choosing discord.