A concise answer: absolutely, it is preferable to define any variable that is strictly internal to your controller as a javascript variable ("var") rather than including it in an Angular scope.
Angular scopes offer numerous convenient features that enable them to function as the model in a model-view-(MV-) architecture (such as data binding). As stated in the Angular guide on scopes, "Scope serves as the connection between application controller and the view".
The wise approach is to only place objects that are essential to your model, which need to be bound to both the DOM and your controller/services/..., on the scope since these features do come with a performance overhead, as you rightly mentioned. Additionally, cluttering your scopes with variables that are not actually part of your model can confuse other individuals who review your code.
Here are some key features of scopes from the Angular scope documentation:
Scopes provide APIs ($watch) for monitoring model changes.
Scopes offer APIs ($apply) for propagating model modifications throughout the system into the view from external sources outside of the "Angular realm" (controllers, services, Angular event handlers).
Scopes can be nested to restrict access to the properties of application components while granting access to shared model properties. Nested scopes are either "child scopes" or "isolate scopes", with a "child scope" inheriting properties from its parent scope in a prototypical manner, while an "isolate scope" does not. Refer to isolated scopes for further details.
Scopes provide a context against which expressions are evaluated. For example, an expression like {{username}} holds no meaning unless it is evaluated within a specific scope that defines the username property.