I'm pretty sure it's been discussed here before.
It is a pretty common problem. Your electric door lock "actuators" are failing.
It will just get worse over time.
If it is quiet and you listen closely, you probably will hear it TRY to move the lock but fail.
The really BAD news is the price Toyota will charge you to fix it.
You might want to consider just using the manual key.