Back in a day, matches were played with one ball which had to be returned from the crowd. Now there are balls everywhere, so the ball is back in play in a second or two. This in itself will have added playing time as only in extreme circumstances would time have been added.
But the principle of adding time when deliberately wasted when the ball is out of play, be it for injuries, goal-kicks, free-kicks or throw-ins is, I would have thought, already within the remit of the referee. It may just be that they are making a much more conscious effort to record that time. And one of the halves of a game today had just one minute of added time ...
But if the point is, yes, let's be stricter on adding time for stoppages, generally I'd agree. As others have just said, unless we are a goal ahead...