You need to think about what a backdoor looks like for different devices, and different functions of that device. “Backdoor” generally means a way to bypass security measures, but that entails can vary wildly in different contexts. For some things you can know because you can check to see if the hardware is doing what’s expected because the only meaningful backdoor would be local to the hardware.
For example, hardware based encryption systems can have their outputs compared against a trusted implementation of the same algorithm.

For cases where there isn’t an objective source of truth for “proper functioning”, or where complex inputs are accepted and either produce a simple answer (access granted/denied), or a complex behavior (logging login attempts and network calls are always expected) it can be harder to the point of impossibility to know that what’s being done is correct.
This is also the case for bugs, so it can actually be unclear if something is a backdoor or an error.
“Any sufficiently hair brained programming error is indistinguishable from an attack by a nation state threat actor”. (the goto fail bug is a great example of this. extremely dumb error every programmer has made, or a very well executed and sophisticated attack.

Ultimately, any system can be compromised by a sufficiently determined attacker. Security cannot be perfect, because at some point you need to trust someone.
The key is to decide how much you trust each system to handle whatever you need it to handle.
I trust my phone’s manufacturer as much or more than I trust the network provider. If I’m doing something naughty the person I’m communicating with getting snagged leads to me via the network and their device without needing to compromise my hardware. I choose to focus on the weak link: the people I talk with who might be unable to properly conduct a criminal conspiracy, and getting them up to speed.