ability to manage access to healthcare records amongst unrelated providers

I have a HDHP with a BCBS licensee somewhere in the US.

I get my primary care through “university A” provider/regional health plan. Recently I went to a “university A” specialist recommended by my primary care for screening of a suspected cancer. I think he was dismissive, rushed, and did a ridiculously terrible job. (It’s not all in my head – a friend who used to work in allied health agrees w/me and couldn’t believe it. Said it sounded like a scene out of a Seinfeld episode!) I now wish to go to a “university B” specialist for a second opinion.

However, years ago I had an issue requiring a minor surgery where I went directly to “university B”. (wish I had in this case, but I digress) As I was being examined by the intake nurse, it was as though she had been given access to my entire medical history – via insurance, somehow? – without me explicitly granting my consent. Mind you, when I called in to sign up for the appointment, I had of course said to university B phone agent, “I go to a university A regional office for primary care”, but she didn’t say anything like “is it ok if we contact them for your medical history”. They are in the same region, so of course would presumably normally share such information.

I don’t WANT the “university B” specialist, this time, to know I’m seeking a second opinion. By having any health plan in the US nowadays, are you de facto consenting to this level of information sharing? Is there any way to prevent it? If I tell university B this time “I don’t have a primary care” – my plan allows me to see specialists without referral – will the insurance still just give them access to my university A health records?

* – it will be a separate physicians practice because it’s a different region of the body, but I doubt that matters from the POV of their records system – the 2018 encounter, showing who my university A primary care was, might still be in there, along with the attendant reciprocal information sharing agreement. It’s almost starting to seem like I need to find ‘university/practice group C’ for the second opinion!